Faculty discussion of online courses at UCSC

This afternoon I attended a Faculty Senate panel discussion on the future of on-line courses at UCSC. A couple of the panelists had already taught on-line courses, and their presentations were particularly interesting.

One had taught a hybrid course where half the students attended live lectures and the other half watched videos of the lectures. Both halves had required weekly hands-on discussion sections, so the course wouldn’t scale to MOOC sizes. The bottom line was that there was no significant difference in performance between the on-line and live-class halves of the class, and that students spent a lot less time looking at the videos than predicted. (The class has been offered 4 times to about 300 students each time, so this was not a small sample.)

The other professor is currently teaching a tiny boutique class (14 students, I think he said), using software that lets him lecture from his office, with a whiteboard window, a little web-cam video feed, and a chat window. I’ve used similar software in conversations with the Global Physics Department (whose meeting tonight I missed, because of the panel discussion, but they were just discussing the College Board’s plan to split AP Physics B into two courses, which I’m not all that interested in). When I gave a presentation to the GPD, I found it very difficult to present material on the whiteboard, talk, and watch the chat box all at the same time. I asked the professor about this problem at the reception afterwards, and he said that with a small, quiet class, he can usually keep up, but if everyone chats at once, stuff scrolls off screen before he can read it. He thinks that the technology might scale up to 60 students with a very non-interactive lecture style and sleeping students (I exaggerate his description), but not beyond that.

Another professor presented a course that is going to be offered soon that takes the form of a self-paced e-book (on calculus). He showed a couple of features of the e-book, and I think that it has many of the bells and whistles that math bloggers have expressed an interest in seeing in math e-books. Personally, I did not find the examples he showed very appealing, but I’m not part of the target audience. (He also loves math history, which I have always found to be a tedious addition to math books, so I’m really not part of the target audience.)

Some of the panelists just raised questions for us to think about, though they went by so fast that I don’t think anyone in the audience will remember more than one or two of them—the questions they were thinking about before coming to the meeting. I hope that the Committee on Teaching or the Committee on Educational Policy will send out the list of questions as e-mail.

One thing that disturbed me about this meeting was the average age of the attendees. I think I was well below the median age there, and I’m turning 58 this week. If we are talking about the future of online education at the university, then we absolutely need to be talking with the people who will be the faculty in that future. It can’t be only us old farts who will retire in the next decade (and the professors emeriti, who have already retired)—where were the assistant and associate professors? I’d be very surprised if there were more than 4 assistant or 6 associate professors there.

My personal feeling is that UCSC should not invest large amounts of money in online education. It does not seem to be much cheaper than conventional teaching methods, and UC does not have a good track record for providing infrastructure cheaply, nor for running businesses. I think that UCSC should be concentrating its shrinking resources on the things where there is enormous value added by being a UC: on lab courses and small seminar courses where students get direct hands-on experience and interact with faculty. If this means outsourcing the teaching of the 1000 students a year taking precalculus, well, that’s too bad, but high schools and community colleges can teach those courses ok. I don’t believe that UC should be teaching precalc—certainly not to a quarter of each incoming cohort!

Unfortunately, the budgetary pressure in recent years has been towards eliminating small grad courses and expensive-to-teach lab courses, and creating more and more mega-lecture courses. These mega-lecture courses are relatively easy to replace with MOOCs, since the teaching in mega-lectures has already been degraded almost to the level of video lectures, with no interaction for most students. Once you start moving to a factory model of education, it starts becoming “obvious” to outsource the production to cheaper labor elsewhere, or to look for “economies of scale” that allow you to mass-produce a course. I’m not convinced that there are economies of scale in education—I don’t think that it is really more cost-effective to teach 1000 students at once than 20 students at once. You can make the course cheaper per student, but the cost in quality is pretty high.

The calculus e-book looks like a promising alternative to big lecture courses, though I suspect that not that many students will slog through it without someone holding their hands and cheering them on. Even my son, who is very interested in math and quite good at it, finds it much easier to learn in the context of a class with regular meetings and feedback from the teachers than in a self-paced course with the same content—lack of time-management skills ruins self-paced courses for most students. Of course, there is no reason that e-book has to be used in a self-paced course, but adding math coaches or teaching assistants to the course raises the cost of offering it to nearly the levels of a conventional course. Furthermore, the time, money, and effort involved in creating such an e-book means it is unlikely that UCSC will create many such resources.

The chair of the Committee on Educational Policy suggested that there would be a market for on-line courses in bioinformatics from UCSC, since UCSC is an acknowledged world leader in bioinformatics. And it is true that there might be a market, but as the teacher for our core graduate bioinformatics course, I don’t think that our quality of education would survive a transfer to on-line format.

My “lectures” are very interactive—I try to get students to derive things like the Smith-Waterman algorithm and the forward-backward algorithm for HMMs from reasoning about how to break problems into sub-problems for dynamic programing. I could present the algorithms in a textbook-like way in a quarter the time, eliminating the long waits for students to digest and idea and suggest a next step, eliminating the cold calls, eliminating the checks for understanding at every key point, … . I can teach a group of 20 students in the same room with me, but I’d lose most of the useful feedback in an online setting. I’d also lose the chats with students between classes—e-mail and forums do not bring up the same issues that come up when I stop by the grad office to get more hot water for my tea. Even recording my extemporaneous presentations would flatten them—I’m likely to be just enough nervous about making mistakes on camera that I’d play it safer, doing pre-canned examples, rather than riskier live-action math and algorithms that show how I think about problems, rather than just showing “the solution”.

Just Monday, when I was presenting a numeric example of computing HMM probabilities, I made a serious mistake that amounted to multiplying by two transition probabilities instead of just one in the first step. It was caught by one of the students, and I could correct it and go on. Today, after we together derived the more general recurrence relation for the forward algorithm, one student suggested an optimization that wouldn’t quite work, and I could point out that it was exactly the same as the mistake I had made near the end of Monday’s lecture. With an online course, either the mistake wouldn’t have happened in the first place (if I polished my examples before presenting them, following a script rather than extemporizing), or the students would not have had the involvement to correct me or to propose optimizations that didn’t quite work. Having a small class that has been encouraged to present ideas, to challenge me when I may be making a mistake, and to ask questions when they don’t understand is crucial to my teaching style, and having a record of the class is likely to ruin that.

I sometimes deliberately make mistakes and hope for the students to catch them—if they don’t, I have to spend more time stepping them through the pitfall, so that they can see it and avoid making the same mistake themselves. At the beginning of the quarter, the students were pretty shy about saying anything, but I now have over half the class participating on a regular basis, and even the weaker students are willing to ask about potential errors, though they ask more timidly than the stronger students, since there is a bigger chance that they are misunderstanding something, rather than pointing out my error. Encouraging the students to correct my mistakes does get me more feedback about misunderstandings, when their attempts to correct something that is actually already correct highlights where they did not quite grasp a concept.

Even if we could somehow magically provide online all the visual cues and social interaction of the face-to-face classroom, I don’t think that we could scale up other aspects of the course: I’m already spending almost all my weekends providing detailed feedback on programs and papers for a class of around 16 students. If we scaled the class up by even a factor of 2, we’d lose that detailed feedback, which I see as an essential part of the homework. For many of the seniors and grad students, my reading of their programs and papers is the first time any professor has read any of their work closely—and they desperately need to hear how to fix their in-program documentation or how to reorganize their sentences to avoid flow problems.

Incidentally, in my other class (which includes many of the same first-year grad students), the students just finished doing 10-minute presentations on techniques from Teach Like a Champion. Tomorrow, before we start reviewing the video recordings of their presentations, I think I’ll have them try to think about which of the techniques they presented that they have seen me use in the bioinformatics core course. This year they presented Circulate, Ratio, Cold Call, the Hook, Pepper, Warm/Strict, Wait Time, No Warnings, Check for Understanding, Stretch It, Positive Framing, No Opt Out, Board=Paper, Call and Response, and Begin at the End. I think that they’ll find that I use about half of those on a regular basis. (I leave it to my readers to guess which of these I don’t use much—those who had me as an instructor a decade or more ago might make different guesses than those who’ve had me recently, as I’ve gotten better about some things.) Note that most of the teaching techniques in Teach Like a Champion are difficult to apply in an online course.

I’m not planning to teach any on-line courses in the near future, and I’ll be putting my efforts into creating more of the interactive, lab-style courses that are difficult to replicate on-line (like the Applied Circuits course I’ve been designing for the past 5 months). I think that the future of the university is in these high-interaction-level courses—artisanal education, not mass-produced factory education. There will undoubtedly be a huge market for the Wal-marts of education, but that’s not where I want to work, nor where I want my son to be a student.

Related

I don’t know. The teacher was Diane Gifford-Gonzalez, and she said she had a detailed report she’d be willing to send to Senate faculty who were interested, so you could contact her. She did show a shift of the course from mostly in-person to mostly on-line over 4 years, but I didn’t catch whether this was a change in student choices or a change in how much capacity was allocated for each section.

I am almost certain it was voluntary. The PBS folks and the EGRs were heavily represented in the on-line version and the anthropology majors (it was an anthro course) were heavily represented in the classical version

The vast majority of studies either don’t say how the students were chosen or don’t make a baseline measurement of the student’s ability before the class is offered or both. The very few that do this find evidence that the students who choose the online course are more confident and have higher scores than the ones who take the traditional class (I will find one such study and post it her, if you are interested). This is important when they finally conclude that there’s no difference in the a posteriori scores of the two groups of students; it leads to an important fallacy. In one study they use the term “do no harm test” or something to that effect, that reinforces the suggestion that the performance of the students is not affected by the format of the course (http://oli.cmu.edu/wp-oli/wp-content/uploads/2012/05/Lovett_2008_Statistics_Accelerated_Learning_Study.pdf).
I also suspect there are other important biases: like Hawthorne effect and the fact the researchers typically have a lot at stake in the outcomes of the study.
By the way, studies do see to find a difference in the a posteriori tests for hybrid courses (the ones that mix traditional delivery with online content).

Because of this, I personally urge caution going into online education (especially if it involves the university going into debt and especially if there is no conclusive evidence about its effectiveness). I also urge people to not mix the financial/accounting arguments with the quality of teaching arguments. Finally, I would hope that we do not adopt “do no harm”-kind of measurements in the effectiveness of our teaching!