Monday, September 10, 2012

Udacity Statistics 101

The prospect of massive-scale online schooling seems to be all the rage
at the moment. Recent competing initiatives include Khan Academy,
OpenCourseWare, Udacity, Coursera, and edX (the latter ones sponsored
by top-name schools such Stanford, Harvard, or MIT, or else founded by
ex-faculty members). The idea of universal and free access to college
programs from top researchers has fired the imagination of many in
the blogosphere, and some have predicted the imminent collapse of
traditional universities in the face of this “tsunami”.

As a
college educator myself, I felt compelled to survey one of these
courses, so as to assess their general quality, advantages, and
disadvantages. (Perhaps there would be some techniques that I could
fold into my own courses.) This summer, Sebastian Thrun's Udacity
unveiled a new course, Introduction to Statistics, taught by Thrun
himself, which I felt would be ideal for my purposes – my current
job largely specializing in teaching statistics at one of the
community colleges in the City University of New York (and my
master's degree being in Mathematics & Statistics). Having
enrolled, I proceeded through the entirety of the course, watching
all of the lecture videos and taking all of the web-based quizzes and
the final exam.

In
brief, here is my overall assessment: the course is amazingly,
shockingly awful. It is poorly structured; it evidences an almost
complete lack of planning for the lectures; it routinely fails to
properly define or use standard terms or notation; it necessitates
occasional massive gaps where “magic” happens; and it results in
nonstandard computations that would not be accepted in normal
statistical work. In surveying the course, some nights I personally
got seriously depressed at the notion that this might be standard
fare for the college lectures encountered by most students during
their academic careers.

Below I
will try to pick out a “Top 10” list of problems with the course.
These are not comprehensive, but I feel that they do give a basic
sense for the issues involved.

1. Lack of Planning

Generally,
the lectures and the overall sequence feel like they haven't been
planned out in advance (and as a result, they don't connect together very
well). One lecture is interrupted by a visitor walking into
Thrun's office as he records it, and this is left in the video itself
(Unit 17.8). Other lectures use a data set of students' guesses about
Thrun's weight for a hypothesis test on
his actual weight – which, not being a population parameter, is
totally incorrect and “an abuse” (as he admits himself in Unit
32.1); yet this semi-accidental data set was convenient to
access, and so was apparently considered acceptable.

But probably the best example of the lack of planning is how radically off-syllabus the
course went from its initial advertising. Now, I've taught courses where things didn't go
entirely according to plan – maybe a lecture went a half-day long,
but never in all my years of teaching has a course so massively
diverged from the initial plan or course description. Below you can compare the starting advertised syllabus (before any lectures were posted) to the revised final syllabus (after the lectures were actually produced). You'll see that they are remarkably different.

Initial syllabus:

Visualizing relationships in data – Seeing relationships in data
and predicting based on them; dealing with noise

2. Sloppy Writing

Now,
I've become fairly “religious” about the text of mathematics –
reading the details correctly, and writing with precision, being absolutely paramount. (And I've found that for my remedial students,
this fairly simple-sounding skill is a nearly insurmountable
stumbling block.) When I saw the Udacity interface, I was initially
excited; instead of a lecturer standing in front of a chalkboard, the
frame is focused on the writing surface, which gives us the
opportunity to highlight and be careful about the writing (this
being similar to Khan Academy, etc.) But soon I became keenly
disappointed at how poor and unclear the written presentation was.

There are
at least two related issues. The first is that new terms and symbols
are almost never given written definitions. Personally, I find that
discussions and questions usually return to the definitions of
terms, so setting those out carefully is the first and most important
task. Here, new terms are casually described in the audio track, but
they are neither technically careful nor visible to the viewer. I
think this is exacerbated by the course's commitment to not
following any textbook or other written source – after the first
encounter, there is no capacity to search, index, or reference back
to terms or definitions that you might need later on (and this holds
as well for specialized symbols for sums, products, conditionals,
logical operators, etc., that tend to materialize for the first time in
the middle of a problem).

But the
second issue is that the algebraic manipulations themselves are
uniformly sloppy and disjointed; some bits of the work will be
written down, the next bit discussed verbally, then another unrelated
scrap written down, etc. There are unfixed typos in words and
equations. Statements and tables go unlabeled, so when a problem is
done you can't tell from looking at it what the point was. Notation
varies unpredictably: at different points in the course, the symbols
μ,
x-bar, and E(x) are all used for the sample mean without introduction
or warning. Usually formulas are absent until given in summary at the
end of a section, and then disparaged as being “confusing and
complicated” (Unit 9.10) or “really clumsy” (Unit 9.15), which
I think is a great pedagogical loss for learning to read and write
math properly. At one point you get to see the assistant instructor write that “0.1 = 0.06561” (Problem Set 2.6), which to me is an unforgivable, cardinal sin. In many cases one would have to rely on
the discussion forums for a fellow student to present a clear and
complete piece of written math for any of the example problems.

3. Quiz Regime

The
pattern of lectures goes like this: A video nugget of a few minutes
will be shown (perhaps 2-5 minutes), which leads to a web-based quiz
question (prompting for retries until success), and then a brief
video explanation of the answer. In general, I like this idea of frequent questioning and I do
the same thing in my own classes: regular check-ins for myself and my
students that we've successfully communicated the ideas at hand.

But a
couple of things make this wonky here. One is that, obviously, the
communication is not really two-way; neither Thrun nor the system is
really “listening” to take note of when a presentation has misfired and
needs clarification. Another is that the quiz regime timing seems
forced and frequently not at a point when there is really a
legitimate new idea to check in on. I would guess that as much as
half the time a question is actually asked before students
have been given the tools to answer it, being used as a means of
introducing a new section. Things like, “Don't get disturbed if you
don't know the answer” (Unit 1.4), or “I'd be amazed if you got
this correct!” (Unit 9.13), are heard frequently. These kinds of
questions seem inherently unfair and, I can only imagine,
discouraging to many students.

4. Population and Sample

Astoundingly,
the Udacity Introduction to Statistics course manages to go almost
its entire length without ever mentioning or making
any distinction between the population and sample in a study. I say
I'm “astounded” because in my classes (and any one I've surveyed
or looked at), this is the key idea in introductory
inferential statistics. It's the very first thing that is mentioned
in my class (or the book), and it's the very last thing on the last
day, too. It's the entire reason why inferential statistics is
necessary in the first place. In fact, the very word “statistics”
means measures for one (sample) and not the other (population)
– but you'll never learn that from this class.

As a
result, Thrun goes the entire course using the symbols μ
and σ
to indicate the mean and standard deviation of both a random variable
(population) and a limited data set (sample), whereas normally they
indicate only the former. He'll switch between the two essentially
without notice, saying something like “the observed standard
deviation” (Unit 25.3), or “our empirical mean” (Unit 25.4).
The x-bar notation appears late in the course, mid-way through a
problem statement – and then being used to indicate the mean of a
population
in a hypothesis test, which is exactly reversed from normal usage
(Problem Set 5.5). And the customary (unbiased) formula for sample
standard deviation is entirely
missing from the course,
necessitating annotated instructor comments to point out that the
results you get from this class would not be acceptable in any other
venue (Unit 27.3).

5. Normal Curve Calculations

A
similar astounding absence: The entire sequence of Udacity's
Introduction to Statistics passes without ever calculating any values
for normal curves. Again, since the course is committed to being
independent of any outside resource (no textbook, no tables, no
statistical software suite), the result is that calculating
probabilities or values for normal distributions is simply impossible
and never occurs. Students don't have any opportunity to develop an
intuition for normal-curve probabilities. The Empirical Rule (the
68/95/99% rule-of-thumb for standard deviations) is never mentioned.
When the time comes to compute confidence intervals, Thrun is forced
to give the direction, “just multiply this value over here with
1.96 – the magic number!” (Unit 24.19), not having any way to
explain where this comes from, nor even mentioning at the time that
this is specific to a 95% confidence level.

Thrun
spends a surprising amount of time developing the actual formula
for a normal curve, but no calculations are made with it and its
utility in an introductory course is highly questionable. The absence
is doubly weird because at one point he asserts, “That's the
purpose of the normal distribution for the sake of this class... we
just do it for the normal distribution where things are relatively
easy to compute”. (Unit 20.15)

6. CLT Not Explained

Another
bizarre gap: what one would think to be the keystone to inferences
for a mean, the Central Limit Theorem (the fact that the distribution
of possible sample-mean values automatically takes on a normal shape
with large sample size) is never clearly stated, nor its importance
explained. There is an optional programming unit with the name in the
title (Unit 19), which does generate a bell-shaped histogram of a few
thousand randomized sample means, and ends by stating that how this
relates to the Central Limit Theorem will be discussed in the next
unit. The next unit is on the Normal Distribution, but it still
neglects to actually state the CLT, and instead winds up engaging in a
rather baroque discussion to wit, “it's a transition from a discrete
space of finitely many outcomes to a space of infinitely many
outcomes” (Unit 20.14). There's a later point where Thrun says,
“Remember the Central Limit Theorem? Remember what it said?”
(Unit 25.2), and weirdly, this is the first time he actually outright
(if very briefly) states it. This is cursorily tied into how
confidence intervals work (blink and you'll miss it), and also said
to relate to “1.96 the magic number” in an unverifiable way (Unit
25.2-3). It's enormously unclear, and I think a distressing misstep.

7. Bipolar Difficulty

Throughout
the course, lectures and exercises veer rapidly between utterly
trivial and nigh-impossible. I think this is a reflection of the
one-way communication channel, such that Thrun can't have any
awareness of what counts as easy and what counts as hard to the
students. Frequently the “problem sets” at the end of a section
will have work that is dramatically different than anything shown in
the lectures. The first half-dozen units of the class are fairly long and obvious presentations of reading different tables and charts
and linear relationships. Then at some point he switches into a
remarkably difficult “complete the proof” exercise demonstrating
that the sample mean is in fact the correct Maximum Likelihood
Estimator for the population mean (Problem Set 3.1; not that he uses
the terms sample/population) – granted that this is “optional”,
but the course hasn't had any proofs at all to that point, the
overall strategy of the proof isn't declared, and it involves
numerous calculus concepts. Even my graduate text in statistical
inference (Casella/Berger) felt compelled to present and explain that
proof in its entirety. (Later, when he revisits this same exercise
again in Unit 23, Thrun actually does finally explain the technique,
which I presume to be a response to earlier complaints in this
regard.)

Similar
whiplash will be experienced at other points in the course. For
example, one student wrote in the discussion forums for the course
(regarding a different problem), “Questions such as this one and
the one before it 'Many Flips' are counter productive. The previously
explained course material was mostly very smooth and gradual.
Reaching 'Many Flips' felt like crashing into a reinforced concrete
wall.” (Link).
That's a perfect description of what I think the experience will be for many
first-time students.

8. Final Exam Certification

The
course ends with a web-based final exam with 16 questions in the same
vein as the section quizzes that have appeared all along. Upon completion, the student is able to print out a PDF “certificate of
accomplishment” saying that they've taken this course from
Udacity, with one of several success levels (Highest Distinction for
all 16 questions correct, High Distinction 13/16, Accomplishment
10/16, or Completion 8/16).

Now
obviously, a somewhat delicate issue is that this is a completely
worthless, faux-certification for a number of reasons. Obvious ones
would be: (1) Udacity has no accreditation, oversight, or recognition from any
outside body, and (2) the questions are all fixed and the
answers are probably posted somewhere online in full. But even more
importantly, and what really surprised me, was: (3) the fact that you
can re-submit all of your answers as many times as you like until
they are confirmed correct (just like the quizzes; and some are even
multiple-choice). Another would be: (4) the final exam is just remarkably
easy; could this be a response to recent criticisms that only a tiny
percent of students who register for courses like these ever
complete them? If this is a PR problem for Udacity, then obviously
they can reduce the difficulty of a course to whatever level
generates a desired completion rate.

Recently,
the blog “Godel's Lost Letter and P=NP” by Georgia Tech's Richard
Lipton had a lengthy post considering a perceived security problem
with programs like Thrun's at Udacity: namely, that a student could
freely register multiple accounts and keep taking the final exam
until they achieved an acceptable score. But this overlooks the
rather blatant fact that no one need go to such lengths, since the
system already allows you to re-submit each individual exam item as
many times as you like until success. Apparently Thrun's own response
to Lipton's concern was to propose tracking of IP addresses to
identify duplicate students, which bizarrely suggests a complete lack
of awareness of how his own final exams work. (“Well
Thrun told me about it in person when I visited his company this
winter. They also can track IP addresses and they can see what is
going on with their students.”; “Cheating
or Mastering?”, August 21, 2012)

9. Hucksterism

As if
the content-based problems noted above weren't enough, running
throughout Thrun's presentations is a routine, suspiciously hard-sell
call for how stellar the class was and how much you, the viewer, have
learned. Personally, I found this to be both grating and a
thou-dost-protest-too-much
lampshading of the flaws of the course. (You might think that I'm being too harsh,
but puncturing this kind of stuff is, after all, the raison d'être
of the MadMath blog). He says: “You now know a lot about scatter
plots!” (Unit 3.12) (yeah, lots). “Isn't this a lot of fun?
Isn't statistics really great? (Unit 6.16) (surely someone thinks
otherwise). “You are a very capable statistician at this point!”
(Unit 32.12) (hyperbole at best). “When people say this is a
contradiction... just smile [in disagreement] and say you took Sebastian's Stats 101
and you understand.” (Unit 22.5) (yeah, I'll get right on that).

10. Lack of Updates?

Finally,
here's a core a problem that multiplies and exacerbates all the
others. In normal college teaching, a truly dedicated instructor will
go through a never-ending process of constant refinement and
improvement for their courses, based on two-way interaction and
feedback from live students. (I know I do; I've taught my
introductory statistics course several dozen times and I still sit
down and note possible improvements after almost every single class
session.)

So in
theory, any of the problems that I've noted above could be revisited
and fixed on future pass-throughs of the course. But will that happen
at Udacity, or any other massive online academic program? I strongly
suspect not – likely, the entire attraction for someone like Thrun
(and the business case for institutions like his) is to be able to
record basic lectures once and then never have to revisit them again.
Or in other words: All the millions of students using these ventures
will be permanently experiencing the shaky, version-1.0 trial run of
a new course, when the instructor is him- or herself just barely figuring out
how to teach it for the first time, and without the benefit of
two-way feedback or any refinements.

Summary

Based on
my review of the Udacity Introduction to Statistics course, I see
some compelling strategic advantages for live in-class teachers, that
will not be soon washed away by massive online video learning. Chief
among them are the presence of actual two-way communication between
teacher and students, such that the instructor can modify, expand,
and respond to questions when appropriate (in regards to clarity of
presentation, quiz questions, missing pieces, and rationalizing
difficulty levels); and the ability to engage in a cycle of constant
improvements and refinements every time the course is taught by a
dedicated teacher. Also, I feel that written text is ultimately more
useful than videos, being more elegant and precise, easier to search
and index key terms and examples, suffering fewer technical problems,
easier to update, and generally being truer to the form of
mathematical written presentation in the first place. In addition to
these, Thrun's lectures at Udacity have a stunning number of critical
flaws (in regards to planning, sequencing, clarity, writing, and
missing major topics) that leave me amazed if any actual intro-level
student manages to make their way through the whole class.

Perhaps
the upshot here is a restatement of the old saw: “You get what you
pay for.” (Udacity being currently free, with a mission-statement
to remain that way). Or else another: “Don't take a class from a
world-famous researcher, because they don't really have time or
interest for teaching.” Obviously, Sebastian Thrun is not just a
teacher-by-online-video; he's also a Google Vice-President and
Fellow, a Research Professor of Computer Science at Stanford, former
director of the Stanford AI Laboratory, head of teams competing in
DARPA challenges, and leads the development of Google's self-driving
car program. How much time or focus would we expect him to have for a
freshman-level introductory math course? (Not much; in one lecture he
mentions that he's recording at 3AM and compares it to his “day
job” at Google.) Some of these shortcomings may be overcome by a
more dedicated teacher. But others seem endemic to the massive-online
project as a whole, and I suspect that the industry as a whole will
turn out to be an over-inflating bubble that bursts at some point, much
like other internet sensations of the recent past.

136 comments:

I may be taking one or two of these courses in the coming year, but (partly since there is no one who cares whether I finish but me) I may have too much on my plate and drop them. I hope to see in-depth reviews about other courses from other bloggers. I believe some instructors will be more responsible, but we won't know until we see the reviews.

Sue, you're kind and perceptive. :-) I do think other instructors could do a better job; one of the interesting things on Slashdot was several people saying that what I wrote aptly described Thrun's AI class, as well. But the completion rates for any of these massive online courses are realy low (like 1-in-30 from what I read).

These seem to be growing pains - we all hope that the new approaches learn from their mistakes as this inevitable evolution of learning progresses. So many more people on the planet will benefit as we move away from traditional, walled education.

I wouldn't judge all online courses based on udacity.com's statistics course. I've taken about 10 online courses, 5 of which were almost completely automated and the remaining 5 had direct interaction with the professors.

In my experience it boils down to how talented the instructor/professors are and how much time they put into developing the online material. The classes that had some instructor interaction were in general a little bit better, but it was not a guarantee.

Bad instructors are bad, whether or not they put together an online course or a brick and mortar one.

You do come off as quite harsh and because this disruptive technology is designed to make you redundant it makes it difficult to measure the credibility of your complaints vs the obvious personal gain you have by seeing this "tsunami" fail to develop. That said, you've done a good job elucidating the problems. I hope they can be repaired and education will become affordable again. College cost me $72 a semester. As you can imagine, I don't have student loan debt. Compare that to the Times story yesterday discussing the student loan issues our nation faces today. I believe we need this technological revolution to succeed, and for that to happen it needs to be of satisfactory quality.

Late commentator here, but: these sort of criticisms seem to aptly apply through Udacity. Basically, teaching is an art, I don't mean "vs. science" but do mean art; the a lot of "magic" happens between things, problems aren't carefully crafted or tested; definitions are glossed-over and ignored, "this will be good enough" type statements abounds, Wikipedia becomes your best buddy to figure things out--and a whole lot of experimentation; actual analysis and examination of the weaknesses when students interact with the material hasn't been done: it looks like a bunch of knuckle-headed bright-eyed idea-men decided to become teachers with no attention to pedagogy under their belts and a whole lot of can-do "they're stupid, let's keep it simple." They constantly have the problem of self-grounding, i.e. make statements a lot of folks won't understand...but actually, as far as programming is concerned, this isn't all bad: it means you're introduced to the kind of thing that's constant in programming anyway; have to become a researcher--and get thrown to the wild and wolves constantly. You try and try to duplicate something just done on the screen and failed, and have to look-up shady quarter-done documentation and curse a damn interpreter, compiler, or something or else; get haunted when you DL Python or Java and then realize yours is a different version and that changes with FRACK your code, and oh yeah, tiny details that weren't really explained well somehow prevent you from doing anything: also, intuitive problem-solving of simple step-by-step procedures you have to magically come-up-with confronts you the whole time...just like it would in the real world.........In other words, the slipshod-approach isn't half-bad for the programming/comp sci side, and that's not a bad thing: it produces technicians rather than professors with no creative ability. In stats and similar subjects...at least in the procedures as laid-out before (unless you're becoming a theorist of stats) not so much. The actual interpretation of those results, of course, is far more complicated, philosophical, fraught with difficulties, etc., than any modern stats course I know lets-on or dares broach.

I think you confuse the problems of one bad online course with inherent problems with all courses. All the way back in the late 90s I took an online course and it had almost none of these flaws. The issue, sadly, is the instructor, not the overall approach. Indeed, online classes can be much more interactive, when done right, because each student can get content customized to their current level of understanding. Monolithic per-recorded lectures don't lend themselves to this, of course, but not only is this possible but it's been done.

The sad fact is that I have taken "in person" classes that were just as bad as the description of this online one. I suspect that the author of the angrymath posting could design a reasonably good online class.

There's a tremendous load of carping from some of the commenters that deserves sending straight to the round file.

Here is a actual statistics professor saying that online course could be great, but this course demonstrated that even with a brilliant person leading it, the presentation of materials and coverage of the proposed syllabus was sub-standard. It doesn't matter whether your college had courses that sucked - but at least you could ask questions of the teacher and reference a standard text, neither of which was available as an option.

If I'd have taken this course (and I'm glad I didn't) I'd have been mad as hell that so much of my time had been wasted with substandard presentation, and even more important, that it has massive holes in the assessment which meant that people could cheat their way through it easily.

What value would you give to a person who said he'd passed this course? None.

I suspect that Sebastian Thrun would find this assessment far more useful and constructive than many people saying "YUO SUCK!"

I also think (like the author of this blog) that online courses are very much the future for higher education. But we're not there yet.

A statistics professor using a sample size of one to generalize about all online courses. I would also accuse him of selection bias, as this is only one of several dozen courses on UDACity, and it is quite likely that he picked the wrong one. For a statistics professor, he is very clearly avoiding basic statistical theory.

For the record, the reasons I picked this one were: (a) it's my subject-area specialty (the one I felt the most qualified to assess), and (b) it's taught by Sebastian Thrun himself (who's gotten gobs of press for founding Udacity and claiming that the experience shames live classes).

I see several themes in this review. First, it seems that whether or not Thrun can use statistics effectively, he's not good at teaching it. If one bad example damned an entire genre of education, then those of us who teach traditional college lectures would be in a lot of trouble.

Secondly, it points out an issue which seems obvious in hindsight, at least to anyone who has developed a course from scratch - it's hard to do, and perhaps impossible to do well without feedback from students as the class progresses.

Finally, there's the issue of reliably certifying someone's mastery of a course. Should this even be a goal? (for high-reputation schools it isn't) If so, there's a known solution - private testing services, as used for e.g. the GRE, MCSE, etc - and the question is really whether there are any easier ways to do it.

I think you've hit this right on the head. Instead of pretending that passing the course means anything, let Sebastian write and teach an MMO course with the intention of passing a real-world certified exam.

THEN we'd have a clear idea as to whether the MMO paradigm (or Sebastian Thrun for that matter) actually makes sense in the real world of qualifications.

That's the nice part about the comp-sci and programming/computing stuff: your exam will be whether you can produce something that lets you either become employed or self-employ. :D ...meritocracy xO ahhhhh

You list 5 online classes and seem to focus on the worst one. Seems your example might be a bit select and biasing. Most of your observations could be duplicated in a live class with a bad instructor as easily as an online course. I remember picking and choosing my college classes based on instructor for many courses.

Is this a current fault? Yes. Is this undesirable? Yes. Then design a feature, a feedback system that works well. Why have opinion blogs by professors, instead, just make them sit on a review/feedback committee.

It's always easier to improve upon something than to start it from scratch.

Indeed, it is better to improve on something than to start from scratch, and this, perhaps, is the core problem with Thrun's approach.

The UK's Open University has been delivering distance learning for 40 years and Thrun's course has ignored almost half a century of excellence in education and gone for a naive ground-up approach to developing a course. The OU over the years increasingly abandoned the traditional lecture format, as it was never really suited to the distance mode. Instead, they replaced them with professionally produced and edited subject-matter documentaries.

Thrun's original claim with the Stanford AI course was that it was that the AI was both being taught and being used to adapt the course.

It was supposed to be something new.

But there's no sign of that adaptive AI in the course delivery -- he appears to have abandoned it and fallen back on an outdated model of distance education.

Your review of the online class very accurate. However, I would like to suggest that - with the exception of accreditation - you will find many similar issues at local universities and colleges. Here are just a few of the ones I have encountered at multiple accredited four-year universities:

1. Incorrect material - instructors making flat out wrong statements related to the course material based on personal experience rather than an in-depth knowledge of the subject matter.

2. Poor command of the English language - You think sloppy writing is a problem? Try a language barrier so high that the majority of students cannot comprehend the instructor's lectures.

3. Errors in assignments and/or exams - sometimes to the point of material from entirely different course content. This is particularly true of "online" courses offered by the universities, but also on person ones.

4. Lack of availability from instructors - maybe you're one of the good professors, but many limit their availability to a few short hours of the week, where students are quickly processed assembly-line style.

5. Lack of feedback - I'm actually of the opinion that I have received better feedback from my "online" courses than the in-person ones. Most homework from in-person courses simple comes back with a grade, leaving it up to me as a student to try to get a few seconds of feedback during limited office hours.

6. Dumbing-down of material - Leaving out difficult, but important concepts simply because students cannot perform at expected levels.

7. The inability to fail students - presumably due to administration.

I think that it would be more accurate to simply say that education in general is in a terrible state, whether online or in-person, whether free, or accredited university. Good instructors like you are the exception, not the norm.

Your Number 2 was my number one problem when I attended Miami University of Ohio.

A vast majority of the professors in the math department were from other countries. That's fine, except very few of them had more than a fleeting grasp of the English language. Every single math course I took was a monumental struggle since the professor simply could not communicate to us.

Matthew: Of course I can't disagree with any of those as possibilities. I've observed at least one other instructor in a class that almost made me want to quit my own job in disgust. A key frustration in my career has been that quality teaching is (crazily) almost a total non-priority for the rewards or promotion at universities, and I wish that would change.

But I was keenly disappointed with the Udacity course, and I expected a lot better given the higher stakes involved. I just can't wrap my head around the idea that putting this clatter-trap on public display to a million people would be acceptable.

I appreciate the time and care you put into this review. The point is that this particular course is not good. It remains to be seen whether online courses can "compete" with classroom courses. Just because something is new does not mean it is better. The bottom line will be whether or not we produce more qualified statisticians or whatever the course is designed for this way or the traditional way, or some other way.

If you need any proof of the worthlessness of the certificate, look no further than me. I did a few lessons of the Statistics before it just got bad (not just "I don't understand this" but "This isn't explained well at all"). I ended up bailing on it and a month or two later just tried the exam problems over and over until I found enough easy ones to pass. I'd have been lucky to get a 30% or so on a real Statistics final exam in a brick-and-mortar college.

All that said, I really enjoyed the Intro to CompSci (really Python) and Intro to Physics class, though they have the same issue of static exams.

I had the suspicion these courses had troubles like this. Though I am a little surprised that the course is so non-interactive and unplanned. Both of those sound very addressable, though maybe expensive for a no-fee non-profit. I am imagining a course where the lecture+questions combination comes out and the professor immediately starts seeing a summary report of student difficulty -- you know, how many tries it takes to get the question right or what-not. Then of course it is up to the professor to decide how this will affect future lectures.

However, I have a different potential defense of this course. Did you have the impression that there were a large number of students active in the forums? Were the students explaining things to eachother successfully? Did it seem to you like students actually had learned something? Your rant makes it pretty clear they may have done this IN SPITE of the professor, not BECAUSE of him, but that is, in fact, a distinction without a difference...

One of the things about our new internet faux-reality is that nothing is ever even remotely separable from its on-line context. So when you say that the course uses no external resource (textbook), that's actually factually incorrect. They may not link to it from udacity.org but it is certain that a large fraction of the students who take the course wind up using wikipedia or "the googles" for further investigation, especially definitions. And you suggested that some of the shortcomings of the course were (at least somewhat) ameliorated by discussions in the forums. It doesn't have no text, it has EVERY text. A pretty minor distinction in some regards, but a whopper in others.

Not saying it's the end-all be-all, it just means that their availability is better than non-availability for some students. "better than nothing"

Good question: I did briefly browse the discussion forums a few times, not heavily (there's one quote from it up in the blog post). There were some people spending a lot of time trying to clarify stuff, but in my opinion the student explanations were really super-shaky (c.f. "a little knowledge is a dangerous thing"). I have a hard time seeing how, or any examples of, someone truly qualified to answer tough questions also being a student in the class (excepting a case like my own).

And one problem is that Thrun's usage of terms was at many points exactly opposite standard usage (mu, sigma, confidence interval, significance in a hypothesis test, etc.), so if looking at a non-related source for help you'd have to know enough to come back and translate Thrun to the reverse of what he literally said (in places).

Part of what was on my mind was Udacity's own FAQ which says, "Are there course textbooks? There are no required textbooks for Udacity courses, and the course content does not follow any textbook." http://www.udacity.com/wiki/FAQ#are-there-course-textbooks

What do you suggest? Does a thousand good courses make Udacity's stats course any better?

There are lessons to be learned from this one course that lots and lots of other attempts to teach lots of students could learn from. I read the blog post as a reality check on both online MMO courses AND the author's own teaching could and should be focussed.

Thrun should be humbled and ashamed by what he's put out as a course. Would he have become what he's become if he'd have started with his own course?

Steve Jobs understood that speeding up the boot time of the original mac from 9 seconds to 8 seconds would save millions of people-seconds every year. It's a big deal. It's worth the effort. So if you are going to have 10X, 100X or 1000X as many people see your lecture than is typical (say 30 students typical) then your content should be better by the same magnitude and you should spend the time to make it that way.

Very thorough feedback. I started Thrun's AI course last year when it was done through Stanford and found it equally as awful. So, it's more likely Thrun (and other UDACity courses are similar). From what I've seen the UdaCity courses are not anywhere near a college level class (and maybe that's not their goal).

I'd be interested in hearing your thoughts on one of the others. I've done a couple of courses on Coursera and found them much better and closer to a true college level class.

That's maybe the most interesting lesson I'm getting from this discussion (I'm seeing similar references to Thrun's AI course over in the Slashdot discussion). I can easily imagine a better-run course; I do wish I had time to review a wider selection.

The strongest point of online education is that the very best version of any course can be copied infinitely and the bad versions ignored. Thus what counts is the best, not the worst. And if the the very best online course is better than the average traditional course, online is the clear winner.

Yes indeed. Ultimately there will be truly GREAT online courses, that can be used by many millions of students. The best of the best. I've had many lecture classes that were quite excellent, and I was a bit sad that so few students could benefit from them. What a waste. I've taken a couple of courses from Sebastian Thrun, including his stats class, and I must say I thoroughly enjoyed them all. The discussions frequently allow one to delve deeper into many of the topics, so that one could get much out of the course. I do have a good background in mathematics (MS degree), so I didn't run into any problems (I can see this might be a problem for beginners though). I've learned much from the 7 free online courses I've taken, and they all have been quite fun as well. I look forward to taking many more of these classes in the future, and continue to recommend them to all!

Also, stellar textbooks exist. Reasons why I had to buy others? Professors had a personal favorite, professors had written a textbook, and professors wanted to use something "modern".

A stellar textbook from the 50's won't be used because its language is archaic, or it doesn't use the very latest bits in a handful of places, or it states that certain things are not solved problems (that have become solved problems).

I have found textbooks even a century old that explain better than a textbook written in the last decade and used them to great success in math classes. But, nobody is going to update old textbooks, probably because of copyright and the inaccessibility of design.

We can theoretically open up online courses to being "managed" open source projects where the "canonical" representation is strictly managed, but submitters have open access to the building blocks to fix errata, suggest changes, etc, and everyone else is "free to fork".

Textbooks severely suffer from having to be pretty much written and edited by a small number of people and the medium is fully published.

"We can theoretically open up online courses to being 'managed' open source projects..."

I could at least imagine such a thing for open-source textbooks (and I've hoped, and I've searched, for a good one). But video is so much harder to re-shoot and re-edit that I just can't see how that would be feasible for a course anything like Udacity's ST101.

Why is video harder to re-shoot and re-edit than a 1,000 math lecturer's prepping their own lectures?

It is also silly to impose the limitations of the old medium on the new medium. We might carry the conventions over for ease of getting the new medium started for both content producers and content consumers, but lets not limit our imagination to video. I'm thinking full on animated perspective controlled narrated three dimensional media broken down into segments that are more appropriate to the density and complexity of material rather than fit to unit-size like 1 hr video or chapter length. Each one of these segment narrations could be re-scripted, and the highly structured nature of the data may open the way for more convenient submission and evolution.

Austin, Delta's point is that "free to fork" works well with source code. A video is not source code, it's more like object code

Graphics analogy: I can edit a copyright-free .JPEG, but I do not get to directly manipulate the elements that make up the picture. If the JPEG was originally made in Inkscape, and you give me the SVG, then I have the source.

There is no such thing as "open source video" because a script is not the "source code" and acting/presenting cannot be "recompiled" as you go -- each film shoot is a unique and unreplicatable incident.

Those who say AngryMath is generalizing from a sample set of one aren't reading carefully. Two very distinct sets of issues are raised in his post. One set has to do with a particular, poorly constructed class. The other set has to do with *structural* problems in the Udacity courseware approach that are likely to impact *all* classes. As a former professor, I agree with all of his structural concerns. I also agree with the various observations here that classroom teaching can be dismal as well.

AngryMath's core point, in my view, is that whatever problems may exist can't be fixed in a system that runs without feedback or communication. Without these, there is simply no basis for improvement.

While I am sorry that your experience was as horrendous as it was, almost all of your top-10 problems were issues with poor instructing and not the nature of this kind of education. As many of those commenting have already mentioned, most of these problems can and do exist in brick and mortar college classes, where they are more frustrating because the students are paying huge tuition to experience them.

Look at this as a tool. Say that the instructor put more time into preparation, and then modified the lectures as he got feedback on problem areas. Then say that the instructor would flip the class. He would have students watch the lectures on-line at nights and then during class time would address any questions and have the students do problem sets, where he can assist them as they have difficulties. Likewise, a history class could discuss issues rather than passively listen to a lecture.

I hope you make a course of your own! I am biased, as I recently cooperated with a couple of other people to make a course that teaches Pythonhttp://dave-pritchard.net/cscircles.pdfand it's becoming possible to make really great textual, interactive websites in an easy way. I am going to subscribe to your .rss to see what's coming next.

And I would say you shouldn't make a judgment on the platform and the promises of it on the evaluation of a single course from a single provide. Did you succeed in your probability course? I mean, it is obvious if you throw a dice and you get a one, it doesn't mean the dice will always give you a one each time you are throwing it.

The quality varies greatly from a course to another depending on the teacher and his/her level of involvement and preparation. Some underestimate the required work to produce a good course. Some are doing very well. And great researchers doesn't equate great teachers. Overall, the platform is widely under exploited. Many are just stitching together already existing videos without any other form of preparation and brainstorming about the learning process and how they should leverage the platform to deliver a great course. That is unfortunate, but it is part of the process.

I am a strong believer on the MOCC platform, even if some courses were a real deception.

Since you took so many courses, would you mind evaluating them? For example, I took the first two classes and Andrew Ng's lectures were excellent, cohesive and thorough (given the limits of the course) while the AI class was interesting at points but in general, fragmented and flat); I listened to small bits of Cryptography and Quantum Comp and they sounded fast-paced, streamlined yet profound. So I'm interested in your comparison.

With the exception of "Algorithms" from Prof Robert Sedgewick (Princeton), I have taken all of the courses that Achille mentioned. I found the MITx (edX) Electronics course as well as all the Coursera courses of high quality - quite comparable to the quality of a typical course at a traditional university (the one exception was the "Computer Vision" course which had decent lectures but unfortunately there were long delays in producing the lectures, and the professor also did not find the time to produce quizzes/exercises). The Udactiy courses ("Introduction to Artificial Intelligence" which I'll count as Udacity and "Robotics and autonomous car") were of significantly lower quality. I am not surprised by Delta's observations and had similar concerns (lack of planning, bipolar difficulty) myself while taking these courses. So, in summary I can heartily recommend Coursera courses to anyone interested. Keep in mind that the amount of time you'll spend on a course will vary significantly (for example, I spent around 2 hours per week on "Game Theory" and around 8 hours per week on "Probabilistic Graphical Models"). Nowadays Coursera gives an estimate on the hours of effort required per week which is quite useful for planning your time.

Thanks for the detailed and thoughtful review. I think some of the MOOCs are making an effort to popularize education and IMHO sometimes go too far in doing so. This is particularly off-putting to those who like rigor and a more traditional academic format. One of the best things about all the MOOCs coming out is there is an opportunity to differentiate the offerings to satisfy different audiences.

I would be interested in hearing your opinion of Coursera's Statistics One offering if you have had a chance to look at it: https://www.coursera.org/course/stats1

I appreciate the thoughtful review and I'm with all of your criticism right up until #10. Just because v1.0 of a new course on a whole new way of delivering courses is crappy does not mean that v2.0 or v12.0 will be crappy. The education system is learning how to scale to the internet and to assume that any course professor is going to continue to be crappy is, I think, unfair.

There is some selection bias in the responses. Some of the criticism is justified; on-line courses currently lack the easy random retrievability of information (videos are sequential and practically unindexed), and indeed more time should have been spent on well-roundedness, polish and accuracy. However there are things to like, where Udacity seems to lead: 1. consistent look, 2. close-up shots of paper and things at hand (though more "eye contact" would have been good), 3. Relatively interactive canvas (e.g. drawn rectangles becoming fields, although looking very sloppy; python coding on the web), 4. Course from someone who is on the top of the field (..er. I mean the autonomous driving; I liked Andrew Ng also for this reason), 5. (which is the biggest) even an alpha version of lectures that one works through is way superior to on-campus lectures that one does not attend, due to constraints (in brick and mortar schools, one can spend as much time commuting and waiting between lectures as on the lectures themselves, and easily have a less motivated lecturer).

The easy random retrievability of information varies between courses. Those with subtitle text files and/or searchable PDFs of the slides are much better in this respect. I am taking the current offering of the Caltech Learning from Data machine learning course and one of the students created an extensively bookmarked compilation of all 18 of the lecture slides PDFs which helps greatly with searching for information. This course is closely integrated with a textbook by the professor which also enhances information searching/retrieval.

One of my disappointments with the Udacity courses is the lack of written course notes and integration with (preferably optional) textbooks.

As many of the commenters have stressed here, the biggest problem seems to have been the quality of the lecturing,the inadequate pedagogical structure and the poor presentation, not the fact that it was on-line. When I took basic level calculus courses in the early '80s, my University offered the ability to watch the taped lectures in the library. I found this the best boon to my learning, ever. I could stop the lecture at any point and cogitate on what was said. I could replay a particularly difficult topic time after time, and I often did. I can't count the times I've attended in-person lectures where a moment of day dreaming or some mental block completely threw me off the lecture thread and made the rest of my time on the bench a frustrating waste of time. Now, recording back then was a more involved topic so care was taken in choosing an excellent lecturer and having a quality recording done. So, to your point, you get what you pay for. This mostly what this is about, the price for quality and what mostly needs to be figured out. But there is no way that this is not how most education will happen for technical subjects in the near future. The real question is "How will course producers fund the creation of high quality educational content?" Tuition? Advertising? I can't see how yet, but I believe it is inevitable.

I wish there were an investment market for this stuff, because it's one of the few things I'd be very happy to short-sell on (the idea that most education will inevitably by automated in the future). If that were the case I don't see why textbooks didn't wipe out most live teaching.

I took this course and while some of the criticisms are valid I think the reaction is way overblown. I hope and expect that the course will be updated and refined so as to fix the rough spots but overall I was quite pleased with the course as an introduction to statistics. I took a lot of math classes in university (although not Statistics) and despite its flaws this class was more engaging than any of them and problems with poor handwriting, errors, confused presentation, continuity, veering off the syllabus, lack of interactivity, accented English, etc. were rife in many of those classrooms. Easier to fix this course than all those professors.

The reviewer seems to believe there is a fight to the death over whether expensive classroom based education or (admittedly over-hyped) inexpensive online remote education is 'best' and there can be only one. I believe there are many people that for reasons of cost, geography, time, etc. cannot spend years sitting in classrooms (I am 60 years old and disabled). He also believes that education without credentials is worthless. I believe education and knowledge are what have value and credentials only matter for your first job and only then because HR departments are too lazy to properly test and interview applicants themselves.

To be clear, I don't think there need be only method of learning. I do think that a question like "are these on-line based courses going to replace traditional courses?" (posed by president of Stanford) has an answer of "no".

And I don't think that credentials are a must; but I think a play-money certificate is better just skipped (even if the course were otherwise good).

Sound like Thrun stinks as a professor. These problems are likely present in his traditional lectures also. I suspect this is an outlier and your sample of one may not be representative of the population as a whole. You have measured Thrun's skill as a lecturer and not the merit of online courses.

Well firstly, I think people are making a mistake to expect Udacity to compete with college courses. That's not the goal of the project. Obviously an automated mass-production approach is going to be lacking in some areas compared to a one-on-one instructor approach. On-line courses have some inherent advantages as well, however. They can automatically increase in difficulty if designed to do so, etc. Most importantly, a lot of people can take the same class simultaneously for low or no cost. These courses shouldn't be taken instead of college, but as preparation for it, or in addition to it.

Secondly, I have taken classes at Udacity and also Coursera, etc. I find Udacity to be the... least "formal". That's not necessarily an entirely bad thing, but it shows they are not trying at all to be like colleges.

Some of the complaints mentioned here are legitimate, though. For example, in real in-person lectures, interruptions happen. When making one video that will be shown thousands of time, more polish should be put into it.

Although on-line classes have been around for a long time, the style of these free classes is still experimental - and I don't think we should fault people for volunteering their time to experiment and try to help people.

At any rate, you can read positive comments about similar programs (although a bit of PR) at:http://blog.edx.org/

Interesting observations. However, my personal opinion is that the problem might have more to do with Thrun's ill-developed teaching style, and not much to do with online vs in-class. I say this as one who has completed courses by both Andrew Ng and Thrun. Since I have more data points, I think my argument carries some weight ;-)

I can assure you that there's a remarkable difference in the quality of the courses offered by the two. Teaching style matters. Ng for example is a remarkably good teacher, while Thrun displayed all the weaknesses that you have highlighted well. I can't imagine that having Thrun in a physical class is going to improve matters greatly either. So as much as I admire him and his efforts, Thrun needs to work on improving his teaching and preparedness.

Therefore, I would argue that the difference has less to do with online vs in-class, but everything to do with one's ability (acquired or natural) to teach.

While interesting, informative, and clearly stated, none of your supporting arguments directly connect with why we shouldn't have online courses. Each one of them seems to be applicable to lecture courses, including the bit about two way communication. I've known plenty of lecture professors who ignored students to the point it was really one way.

You should refocus your thesis from "online lecture courses are not as good as live lecture courses" to "sebastion thrun's online stat 101 is not as good as my live lecture course" or something similar.

I went trough AI class back then.I also went trough few of Coursera and Udacity courses.From my experience with AI class I can say that many of your observations are correct, but that seems to be Thrun style. To fix some of those I even created a custom interface for AI class http://www.wonderwhy-er.com/ai-class/ (searchable, can read question, or jump to lecture etc, helped a lot with AI class)

I was thinking to do the same for all Udacity courses but going trough few of them found out that its not that much needed in them. Firstly they improved, secondly different lecturers styles and disciplines did not need it as much.

What I can say is that almost all Udacity courses and some of Coursera courses feel to me like a first try with lot of rough edges. But they are improving.Like Coursera SaaS class has a book for the course, and they are releasing improved version based on feedback.

Also from my experience with both Coursera and Udacity courses I can say that so far they feel different. Coursera courses mostly are rethought for online replicas of real university courses. And I like level of quality of content they have.Udacity on other hand seems to do everything from 0. As a result approach to online education is better but quality of content is worse.

One thing where I disagree with you and find that online education can change things is feedback loop.I am actually very enthusiastic about opportunities with online education because now it all goes trough computer system. And all that data is stored and can be analyzed with such precision which is just not possible in real classes. Like say Zynga is doing with their games, constantly, weekly if not daily analyzing their users behavior and rolling out changes, comparing change in user behavior.And Udacity only learns to do it for education. So for not very well it seems.

Another thing is about it being free. You probably did hear of Thrun hopes for business idea. Teach as many people as you can so that they get a job, and then you get payed, they don't get a job, you don't get payed.While I agree that currently Udacity courses will not produce good professionals its what they need to strive for. And that's their real metric of success for which they need to optimize courses.

About certificates, I think that may be online education has potential to provide something better here, they can provide access to student data. Not say "he done well" but show what exactly he done. Show quizzes, show home works, etc. I mean now its all online and potentially easily accessible so that employer can check what exactly was thought and how it was tested.

One more thing to comment on is completion rates. Comparing it to university courses seems to me like comparing eggs to oranges. I have dozen of unfinished courses on those platforms. I actually register to 3x more then I can follow, just to pick ones I like more.And here again I think its a great thing. I would say that Udacity is a great tool to try different courses to pick high education program. I mean most of my generation after school did not knew who they are, who they want to be, what they want to study. Yet they needed to decide what they will spend next what 3-10 years of their life on? And spend lot of money too. In my view its school and university education problem where free online education can fill the gap. Be intermediary and supplementary education.

Anyways my conclusion so far is that its just different. There is a lot of potential, and lot of problems. And I am happy with what I am getting from it.

"One more thing to comment on is completion rates. Comparing it to university courses seems to me like comparing eggs to oranges. I have dozen of unfinished courses on those platforms. I actually register to 3x more then I can follow, just to pick ones I like more."

That's a good point, but it's equally valid to turn it around: Online universities shouldn't be touting massive "enrollment" rates, because it doesn't mean the same thing as traditional schools. With no barrier to entry (free cost), people are "registering" just to peruse the material, and haven't really committed to seeing the course through in the same way yet.

After reading your article and seeing the class syllabus I have no doubt that this is not the typical introduction to applied statistics class that you teach (and I have taught). You have also convinced me that it was done sloppily with little preparation.

When I taught it, the course was a full semester, not 6 weeks plus an exam. I don't recall which textbook I used, but they all covered pretty much the same material in the same sequence. I would think that this course would be a good candidate for online presentation *if* it were done well.

If *you* spent a *lot* of time converting your course for online delivery and used and revised it a couple times, do you think it would be an effective alternative to the *average* face to face stat class? (I am sure there are some really bad face-to-face intro to applied statistics classes going on as I type this).

Now that's a great question, and I think the answer is "yes", or at least clearly better than this one. This would presume a few things: (a) The fact that I get to use my 7-years of refinements, and (b) I would need an associated book because I emphasize careful reading & writing so much (in fact, if it wasn't my current book it would be a step backward because I tie things to it so much).

The remaining problem would be offhand comments and quiz-y questions, which likely would have to go through refinements to see how they work online. And the final stumbling block is that revisions to video are hard and probably don't have a work process.

So: I think I could make one work better than Udacity ST101, but not as well as my current live class. And much worse if one had to be committed to a "no book" or written material policy (as per the general Udacity FAQ).

Astonishingly bad teaching happens every day in classrooms all around the world, and what's worse, we have no idea of when or where. If nothing else, Udacity and other MOOCs provide teaching in a transparent forum, exposed to the criticism necessary to make it better. Hurray for transparency!

I think it's a mistake to compare an online video to an actual course taught face-to-face, and furthermore there does seem to be an ulterior motive here, loath as I am to say so.

Perhaps it's a class war: only the wealthy are supposed to afford college anymore, and the masses are supposed to get their education from poor quality videos on the internet. Probably not, though. It's more likely that for-profit education interests are noticing the exorbitant tuition being charged by non-profit public and private universities and all that easy student-loan money available for education, and now they're getting greedy and moving in for the take.

Now I don't have an objection if somebody truly finds it helpful to watch "instructional videos" of that style, but *I* don't learn that way, and frankly there are plenty of other excellent resources available online if you are a motivated self-starter willing to invest the time and effort to learn a subject such as statistics.

Courses taught face-to-face by professors will always have their place, because let's face it, if you aren't motivated enough to go out on your own and buy a used textbook online, work through that, download some free software like R, find some tutorials on that to work through, test yourself, evaluate any gaps in your knowledge, and keep learning until you feel you have mastered the subject, you just aren't going to succeed if you equate a watching lousy video posted on the internet to taking an actual course taught by a professor.

Here's my situation: I took basic statistics years ago in college, and last year I refreshed myself on the subject. Now I'm in a situation where I need to learn more intermediate concepts such as multiple regression, tests for multicollinearity, the Breusch–Pagan test, the Durbin–Watson test, ARCH models, and so on. That would be great if I could learn this material by just watching a video online and taking a quiz or two, but the truth is that I'm just not going to learn this material except by my own sheer stubbornness.

Then there's advanced statistics where they get into things like the Stein effect and the finer points of frequentist and Bayesian paradigms. I haven't got this far yet, but I'm not sure why they put all this off so far, because it's pretty important to have a good grasp of the statistical paradigm you are working with right from the get-go, if you want to understand, say, what a significance level is.

I was initially skeptical of Udacity, in large part because Ng's Machine Learning class was so much better than Thurn's AI class for Coursera. However, I've changed my tune, and I do think that Udacity is poised to win the MOOC battle.

Coursera is essentially about a putting a college class online. Watch 2 hours of lecture videos, take a short quiz. Udacity, on the other hand, is asking the hard questions about what learning online should look like, and developing a more interactive, personal experience.

Because Udacity is innovating more than the other online courses, the material is sometimes a little rougher. But through their experimentation, they are rapidly converging on a compelling format.

My students *vastly* prefer Udacity to any other online course they have taken.

As long as Udacity continues to iterate and refine, I predict Udacity will succeed gloriously.

Agree on same. Coursera some of the courses are downright awful in delivery style.I just couldn't follow Algorithms,Cryptography and SAAS because of way classes were done. There was no advantage to being online.Recently I joined "Introduction to Computational Finance" and saw a live class recording which I might as well get from Itunes.

When you say, "Udacity is innovating", are you talking about refinements to existing courses, or changed strategies with newer courses they roll out? (I mean, ST101 I think is the newest one so maybe in this case there's no difference).

And when you say, "My students *vastly* prefer Udacity to any other online course they have taken.", are you talking about ST101, or Thrun's teaching on other subjects, or something else?

The entire Udacity system of bite-sized videos and interactive questions is a significant innovation over anything Coursera offers. The quality of the screencasts have improved. The ongoing tinkering with homework, quiz policies, self-pacing and ongoing enrollment is working out well and shows strong insight as to how to cater to people who have other things going on in their lives.

My students prefer all the Udacity courses they've taken to both Coursera and paid, commercial online courses. They had only good things to say about ST101, as well as Thrun's AI for Robotics class, commenting on his warm, friendly manner and infectious enthuiasm about the topics.

I've posted my thoughts in my blog at http://www.mcgurrin.com/robots/?p=120 I agree with most of your specific criticism of the course, but 1) the other Professor Thrun course I took at Udacity was much better (though with some of the same shortcomings) and 2) I think that the tremendous cost difference between what can be offered in an MOOC versus the current cost of college tuition makes a revolution (albeit, not a total transformation, just as TV didn't end radio) inevitable. There's too much value in getting 80% of the value (in a well done course) for 1/10th or less the price.

I took his AI class and dropped halfway.I don't like Sebastian Thrun's style of teaching, it doesn't work for me.I was simultaneously doing DB class by Prof Widom which I did complete. I also took CS101 at Udacity taught by Prof Evans.CS101 was very well done.I think you should try a different course not taught by Thrun in Udacity and see.Its not the online style of education its who is teaching.

It's worth a read. The essence is that Udacity will be "majorly" updating the statistics class in the near term.

I've taken a number of Coursera and Udacity classes. Like others have said, the Coursera classes (in general) have the feel of normal university classes transposed to an online format. Udacity, on the other hand, is really trying something different. They seem to be trying to emulate a tutor/student relationship, with very frequent interactions.

Thanks, you obviously are full qualified and made a lot of effort to make your point that I completely agree.

If I can say something about this, also prof. Thrun is full qualified and made a lot of effort to make online courses, that I completely appreciate.

I think that is one of the first that made possible to everyone to have a fresh start in some difficulty fields, so maybe he can make a lot of error, and criticism like your (qualified and precise) are welcommed and will help him to raise the education level of udacity online courses.

So thanks to both, and to all people making online courses and people making full constructive criticism.

But I have only one last question: why YOU (forgive the unpolitycally correct upperchars) don't make an online statistic course ? If you will ever do that I will follow it with love.

Thanks for the comment. Honestly, it is interesting to think about how an online statistics course could be done better. One thing is that I am deeply rewarded by the human face-to-face interaction of live teaching. Second would be that philosophically I rely so heavily on the written text, and a place like Udacity is institutionally committed to the opposite (no book no written materials).

I also have many reservations about on line courses. I think, however, that in this case your criticisms are rally more of the course itself than the media. Possibly the way on line courses are produced and promoted today does make it easier to do a poor job.

Coming from a similar viewpoint in wanting to sample what MOOC's have to offer, I suffered through the first handful of Thrun's lectures myself. I was constantly wondering when it was going to "really" start, but I ultimately didn't have the time or patience to go as far as you did. I had similar issues with Coursera's Quantum Mechanics and Quantum Computation which had a very "wishy-washy" feel as well. None of the handful of courses I've sampled so far on these platforms has lived up to the quality or content of traditional classes I've ever taken.

In thinking about the updating of courses over time, I will mention that one of my favorite parts of one of the earliest examples of MOOC's was the frequent "retaping" of lectures by Gilbert Strang in his MIT OCW linear algebra lectures. Though the video tape did it's best to hide the fact that he was often lecturing to an empty lecture hall, I always gave him extra credit for going back and retaping full lectures to correct what I can only imagine are what he felt were poor performances or issues he had with his own presentation. That dedication and his natural talent as a lecturer are what make that particular class one of the few lasting classics in this budding genre. (Of course it doesn't hurt that Strang has a very long history of having previously and frequently taught that same course.)

One of the other big issues these courses seem to suffer from is the very broad range of students which many of them are trying to appeal to. Because many seem to be playing a game of "Which course can have the biggest enrollment numbers?" (instead of focusing on quality and completion rates) some less introductory type classes have some exceedingly large foundation issues. Interesting sounding courses like Quantum Mechanics and Quantum Computation are stacking the enrollment numbers by dumbing down the early material and attempting to make things seem easy enough for anyone with a high school education. Thus the background of their enrollees is probably not quite up to actually getting past even the first lecture. Naturally they pay for it a handful of lectures into the course when the material becomes so deep that those without an exceedingly good preparation in advanced mathematics and physics will get lost in the quickly rising tall grass. Like Thrun's statistics course, Coursera's QM & QC class was missing any reasonable textbook, concise definitions, consistent notation and by lecture 7 had probably lost the vast majority of its audience when the level of mathematical maturity jumped from a very basic almost high school level, which the instructor seemed to implicitly insist was fine for the entire course, to that of a third or fourth year college level. Though I haven't seen detailed published attrition numbers throughout any of these courses, one can take an example of the almost exponential attrition rate of IIT's Quantum Mechanics lectures on YouTube http://www.youtube.com/playlist?list=PLDBF155FF78B995F9 to be a rough equivalent - though my guess is that given the publicity MOOC's are getting their drop offs are probably even steeper.

I'm hoping that once Coursera, Udacity, et al have been around for a while, they'll have developed a more full curricula and provide students with advice and assistance in plotting their way through multiple courses to arrive at a particular destination - i.e. they'll have the traditional equivalent of a "freshman adviser" to help you plan out a way to graduate in four years.

Great observations, thank you for those. I had also wondered how anyone can manage prerequisites in a case like this. Usually it takes me years to understand students' median abilities coming into a course and interface correctly with that (and that's just at a single school). Probably the toughest class I teach is one that gets a lot of widely varying skill sets, which seems like these MOOCs would be times several magnitudes.

Fantastic example of the viewership dropoff on those QM lectures, what a terrific example!

Thank you for the great review of the Udacity course. I have been trying a Coursera course myself and, while finding that I am getting some valuable information out of it, the inability to get any feedback beyond what the mechanical grader tells you is quite off putting. I hope that reviews like this will highlight the need for interaction as the current model (at least for Coursera) seems to be that of the "sage on the stage," which, while certainly not uncommon in intro level courses at large universities, is not what I find to be the enjoyable or easy way to learn new things. Though this is, no doubt, a "get what you pay for" issue, I do think that some general moderation of the course discussion forums would vastly improve the user experience.

thank you for your review. I wholeheartedly agree, but deviate at one crucial point:

I think you are sadly over-optimistic about the didactic and general quality of teachers and professors. Maybe you are brilliant both in statistics and in presenting / teaching it, but:

Do you really, really think university professors get, are able to get and are willing to take any feedback? My statistics university professor had great freedom in anything from his department and the university. Fail rates were high in his classes and he was proud of it. Taught was by his smeary handwriting on a backlight projector, live, in front of 400 students (! introductory, huh...), without any prepared notes or anything typed.Scripts were forbidden and those who bought a script (created by students) were threatened to be sued or kicked out of the university hall for owning such a script (even though they just wanted to learn, understand and at least pass).

As university professors in general might not have the obligation (are not benchmarked) or time or interest to improve their courses (not everyone has your basic teaching motivation!) or are self-absorbed with their academic greatness (even if it's only the one in research, but not in teaching!), I doubt students in a "brick-and-mortar" university are better off.

If anything at all, this only shows that professors at Stanford suck as much in terms of didactics, pedagogics and general "teachability" as at any 0815 countryside university.

At least at Udacity, coursera etc. students can leave feedback both with their names or anonymously and can collaborate - do you know how rare real collaboration and help is amongst "brick-and-mortar students"? Especially if they don't know person x, there is no way to collaborate. A course in this format (just that!) allows people to collaborate effectively.

Why is this needed? Well, because also in my (European) university, many course operators simply shut down (closed) the available "forum" option for their course, i.e. in practice even though our university had the technical infrastructure, course professors and their departments undermined this option based on a diverse set of nonsensical reasons.

I am fed up with the elitism, arrogance and "we do whatever we want"-attitude of departments and professors - such a thing would never happen for classes that are online-based AND paid, because bloggers like you would hold professors accountable publicly in a way that is rarely done for the current closed-door, high-price brick-and-mortar university classes.

"I think you are sadly over-optimistic about the didactic and general quality of teachers and professors... Do you really, really think university professors get, are able to get and are willing to take any feedback?"

I'll just say this: In the entirety of my college math schooling, I never had a class as poorly structured or explained as Udacity ST101. (U. Maine 1989-1995).

I strongly dissagree with Delta's approach. I believe Thrun is an excellent teacher. I took the AI course and scored 100% in the final exam which btw found quite challenging. I am a Maths teacher too and I like this new way of teaching Math concepts without unnecessary formalitties. Remember this course (ST101) is not for Maths students. It is about using Statistics in our world and especially in engineering. I also took Andrew Ng's Machine Learning class and I think the material and the proffesor's insight is excellent as well. But Andrew is boring compared to Sebastian :)

Comment from Thomas Strohmann that didn't want to show up:----------------------------------------------------------I am taking the QM&QC course right now and have a very different view. First of all, I didn't notice any inconsistency in the notation, the professor took care to use consistent notation. Also, the course website recommended a list of very useful textbooks on quantum computation. Further, I did not find that the material was dumbed down early on (in fact I got my lowest score on the first assignment). Working through assignments, and yes this does require active effort as opposed to just passively watching videos, is a crucial part in keeping up to speed on the course. Lastly, I want to point out that each Coursera course has a section on "Recommended Background" right on its sign up page which is very helpful in assessing whether a course is right for you or not. See: www.coursera.org/course/qcomp

As I said in a post above, from the more than a dozen Coursera courses I have taken (and completed), I can heartily recommend all of them including QM&QC (the one possible exception is the Computer Vision course - see my comments from above for details).

After taking the Udacity class I got a copy of Weiss Introductory Statistics on your recommendation. I am only a couple chapters into it but am liking the textbook a lot. It fills in the gaps, has lots of examples, is more rigorous but not to the point of being dry, and is even visually appealing. I like learning by reading in any case, at least when the textbook is good.

Still, I liked the Udacity course too, despite its faults. I liked the informal presentation, the brief videos interspersed with questions, and the problems sets. To be honest, I really had almost no interest in statistics before taking the course, I thought it would be about as much fun as accounting (always good to insult two disciplines in one sentence!). I took it mainly to see what taking a MOOC course would be like, but I got caught up in it and wanted to learn more. I wanted a textbook and thanks for the recommendation. I've also taken Udacity's CS101 and PH100 and will be reading textbooks for Python and Physics to supplement those courses.

I would recommend that other students consider that approach too. I would also recommend that Udacity provide short lists of quality textbooks for further reading, at least for those courses where appropriate textbooks exist. I can understand their goal of free education and how that is not compatible with requiring a (in some cases expensive) textbook, but an optional textbook or online resource (often many free ones)seems fine to me. I think an interactive online presentation and a structured, written treatment of the material complement each other nicely.

I'm so glad that helped! The Weiss statistics book really is probably the most enjoyable text that I teach out of. Complete but as concise as the subject allows. One thing I try to do in all my courses is to support both the current and prior edition of a textbook -- granted prices are so high for current-editions, but older editions are nearly as good and immensely cheaper. There's a lot of good that could be done by using back-edition texts like that.

That Wikipedia page is a good one and one of the more understandable ones. The external links at the bottom were nice too.

I did use Wikipedia often during the Udacity course to get a different presentation of a concept or a more precise one. It did require scrolling through the page to find the 10% of the information that actually applied to what I wanted, they were often too thorough for me. The day I can read a Wikipedia math page end to end and understand it all I will be very pleased with myself. For now, sometimes just for yuks I choose one of the foreign language translations from the left of the page and laugh when it doesn't affect my level of comprehension at all.

- This article may be accurate (I don't know statistics enought to judge it) but i really like the teaching. I like the way Sebastia Thrun teaches (I took all it;s online courses).Maybe the question in advance are a little unfair, but i don't think that's a big deal.- i live in romania and i would have been VERY happy to have a course half as good as this... there are oher places in the world when you can;t get even that. online education will make learning available to everione who WANTS to learn. - the accreditation may help you. in your CV will show that you are interested in the subject. It would not have the power of a diploma, but it will count. beside that, the answers probably can be found online, but the asnwers from an real class can't be shared aswell. There's no difference. I have one class where the quiz was 100% identical with one of the last year.- I don't argue that this course couldn't be better. You are probabilly right. But you judge online courses based on only one course and i think you are a little to harsh. I don't think i remember reading something good about it.

Sebastian Thrun is the best ! I am a computer scientist who needed an introduction into Stats. Not only did I utterly enjoy the course, but I could manage to do it in the very limited time between waking up in the morning and my baby waking up. It was free, very useful and I already applied some of it successfully in the real world. Thank you Sebastian and Udacity.

I'm surprised no one has mentioned the Probability and Statistics course from the Open Learning Initiative (OLI) :http://oli.cmu.edu/courses/free-open/statistics-course-details/

The course has been demonstrated through independent controlled experiments to lead students to learn better in half the time of regular courses:http://www.sr.ithaka.org/research-publications/interactive-learning-online-public-universities-evidence-randomized-trialshttp://oli.cmu.edu/get-to-know-oli/see-our-proven-results/

It's serves to counter three of your claims:

>> “You get what you pay for.” (Udacity being currently free, with a mission-statement to remain that way)

OLI is free and funded by grants, administered by with deep understanding of education.

- “Don't take a class from a world-famous researcher, because they don't really have time or interest for teaching.”

While I wouldn't disagree with exact phrasing, people might interpret this as "don't pay attention to learning products from researchers", which I certainly disagree with. OLI is created by researchers, experts in the science of learning. They aren't any good at making self driving cars; they are experts at enabling effective self-paced learning.

>> Some of these shortcomings […] seem endemic to the massive-online project as a whole, and I suspect that the industry as a whole will turn out to be an over-inflating bubble that bursts at some point, much like other internet sensations of the recent past.

Udacity should not be the poster child for massive online learning. There are plenty of more rigorous and effective systems. I suppose they aren't as sexy.

"The course has been demonstrated through independent controlled experiments to lead students to learn better in half the time of regular courses:"

I would just like to point out that the actual conclusion of the independent trial of the OLI statistics course is that student learning is equal (statistically zero difference) to that of traditional live courses. This is about what I've come to expect from other research studies in altered teaching strategies (example).

"We find that learning outcomes are essentially the same—that students in the hybrid format "pay no price” for this mode of instruction in terms of pass rates, final exam scores, and performance on a standardized assessment of statistical literacy. These zero-difference coefficients are precisely estimated. We also conduct speculative cost simulations and find that adopting hybrid models of instruction in large introductory courses have the potential to significantly reduce instructor compensation costs in the long run."

Good point. I conflated the two studies. The study which found equal or better learning in half the time was for CMU students and not independent.

In the independent ITHAKA study across 6 diverse institutions, "results indicate that hybrid-format students took about one-quarter less time to achieve essentially the same learning outcomes as traditional-format students." So only a savings of say, one month in a four-month semester rather than two months.

The summary statement that effective learning technologies will "significantly reduce instructor compensation costs in the long run" seems like a red herring to the question of the quality of online learning and the value it provides to the learner.

This post is a bit unfair (perhaps due to some corporate motivation...). Indeed, the course ST101 is not good (I try and dropped it) and you are right about many of the Thrun's mistakes (the most apparent to me is the "Bipolar Difficulty" which I agree entirely).On the other hand, I attended the Udacity CS101 (with David Evans) and it was superb. I also take a Coursera-Stanford coursera that was great: I was able to complete it and learn many things.So, in 3 courses I did, my experience was positive in 66.66%. I do not think it's right to generalize based on a single example (and just pick a bad way). Also, you said the only negative points, completely omitting any positive aspect of the course. This is not critical thinking but just an attack.

I took the course myself, and found it enjoyable if a bit introductory. While many of your points are fair (we'll see about "lack of updates"!), the overall verdict seems awfully harsh. These items seem more like quibbles to me - some differences in approach are almost certainly due to the fact that he is teaching outside of his prime area of expertise here, and thus probably ends up producing an "Introduction to Statistics as it is useful for doing AI".

The bottom line is this:If you are able to teach 100 students per semester, two semesters per year, over a 20 year teaching career, you will have impacted 4000 students in your lifetime.The Udacity CS 101 Introduction to Computer Science class has already enrolled 200000 students.Sure Udacity has its shortcomings. But who has a more significant impact?

I've posted a review of Udacity's Web Development course on my blog. I specifically wrote it as a companion piece to your stats 101 review as I think it neatly addresses the complaints that you were only looking at one course. I see a lot of similar issues in CS253, and I'm not too impressed with Udacity's chosen path for the future.

Specifically, Thrun, an academic who presumably knows the value of abstract thought, seems to be selling himself short by aiming at a vocational market rather than an academic one, and selling out by going down the vendor-specific route....

Online classes can supplement an instructors capability on any given topic and provide additional means for the student to receive divergent views and further inspire involvement in the topic. As a "Consumers Report" for online classes evolves,maybe from blogs like the above,AND international testing standards gauge the educational results, online education will evolve and grow.

Present education (and medical)costs are creating debts that rival the subprime real estate debacle. Content and delivery of many online courses already exceed the capability of any single instructor or institution.

I can't speak for the Statistics class, but I can for the CS101 course. I've taken a handful of programming courses at a couple of traditional universities. Compared to all of those the CS101 course at Udacity wins hands down as an introduction to Computer Science. My first programming course was absolutely terrible. I got an A in the class but I walked out feeling like I had no clue what I was doing. I didn't have any idea how to solve problems programmatically and that is one of the most important things programmers need to learn. Language syntax is just memorization, but if you can't solve problems then your useless as a programmer. This is the central idea behind the CS101 course, but using programming a web crawler as the vehicle for that. It's also learn by doing, not learn by being talked at or reading. The Udacity CS101 course gives you a concept and then quizes you on that concept, prompting you to think for yourself how to solve the next step. I've done more problem solving in in the first few weeks at Udacity then I did in a couple of semesters at traditional universities.

It does seem like the consensus of comments above is that while Thrun's Statistics and AI courses are equally weak, Evans' CS 101 course is very well-received. So I suppose that highlights the importance of finding the dedicated teacher in question.

Your review sound incredibly biased. Although it may deliver important information about certain deficiencies, it doesn't recognise it strengths. I've already taken a Mathematical Statistics class with a good emphasis on proofs, and while this class seems a little cartoonish at its side, I would have to say that is on of its strengths. Udacity is about the learning experience, and I have to say that there is no better learning experience that the first few sections of Statistics 101. I loved how he portraid the MAIN INTUITIONS about statistical concepts or at least made you think about then in the quizzes.

Is learning the capability of being able to understand formal definitions? Thats a Turing's Machine kind of approach, we humans learn something new using our knowledge of past experience and mainly everyday life, a good pedagogy would take advantage of that and thats what I find in all of Udacity's courses.

Udacity is just starting, its content is far more edited than Courera's, which resembles more to classical lectures in a short format (some are PEDAGOGICALLY TERRIBLE), and this limits the speed at which Udacity can produce new content. In the future they could produce an Advance Statistics course to please your "formal concepts" needs.

I've had teachers that according to your list of values would recibe excellent marks because they knew all about what they were talking about and there definitions were perfect like in a textbook. But they've lacked one thing: pedagogical skills, and pedagogical skills is like a resistance in teaching; it doesn't matter how much the teacher knows, if they dont have this skill, students will not learn and even start to hate the subject. Information Learned = ( P. Skills ) * ( Information Sent by Teacher) or L = P * T, where P = (0,1). If class were about just having high-level formal definitions, we would be much better just reading books.

Okay, so we all agree that the Udacity Statistics 101 course is "cartoonish" (as you say), and sloppy and lacking in precision.

But I find the counter-defense that it is "intuitive" frankly laughable, because at the end of the course you can't even compute a probability for a normal curve or have any awareness of the 68-95-99 rule-of-thumb. That's a catastrophic failure to deliver the most basic intuition of the subject matter.

I totally agree with the article. The pity is that MANY of Udacity courses are like that. I attended to about 6 of those courses but never finished one: they're very badly done, bad explained and often they give you so few examples that you come to the exercise and you don't even know where to start from!!!

Thanks for the feedback -- indeed, the commentary I've gotten from people who took other UDacity courses is almost always identical to my findings above. (I recently had another friend try the HTML5 game programming course -- a former professional game engineer -- and found it to be basically useless and gave up on it.)

Relating to your last point: Dan Ariely (and team) did an absolutely fantabulous job with their first Behavioral Economics class at Coursera - including course-correcting for a lot of feedback along the way. So it is possible for a famous and busy professor to get it right.

What an awful course. I took it on recommendation from a friend for a pre-course statistics test.

Although I did well in the course, I had absolutely no clue what was actually being taught or how to apply it.

I went on to begin studying using some other materials, and found that I could not do basic statistical problems from other sources.

Eventually, I went out and bought the recommended textbook by Weiss at a used bookstore. I pretty much had to go back through the entire textbook to learn statistics. I can't think of one chapter that the udacity course helped with.

The textbook approach took another couple weeks of full-time study, but ultimately I learned basic statistics and can perform all the other good stuff that was missing in the statistics course on udacity. I had to re-learn probability notation in order to use other sources and read about statistics online.

Strangely, the udacity course included alot of calculus-based proofs (I performed them) and some programming; but was lacking an incredible amount (~80%) of basic statistical material! Proofs are absolutely unnecessary to perform basic statistics, and many of the intuitive explanations offered by other sources are way more useful (such as explanations on the 'defining formula' for various statistics.

Thank goodness I did not rely only on the udacity course, I would have failed my pre-test and had to take remedial statistics.

I would recommend that udacity takes down the course. It can cause two problems:

A) Mislead someone into using it to study for a test (my case). This is a very serious issue if a deadline or exam date is looming.

B) Waste someone's time that could be spent studying other topics.

I am sure Sebastian is a smart guy and great teacher. But it does feel like this course was done 'on the fly' without any planning or direction. It really is a 'topics in statistics' course more than anything. Most likely he was demonstrating the technology rather than trying to properly teach statistics.

The udacity format IS a good format, it is a matter of taking the time to utilize it properly. I think the issue boils down to the basic premise of all classes - the instruction is key. A good, patient, and motivated, instructor is what really counts.

Thanks so much for sharing your experiences! I think that's incredibly valuable feedback to hear from the perspective of a student actually needing to learn the subject (including standardized notation, etc., to apply in other venues). I'm so glad that the Weiss text could help you there. Appreciate the comment very much!