Category Archives: Placeholders

Been doing all sorts of crazy things with the main texts I’ll be using in my SOCI/ANTH 441 Material Culture course.

(Not sure how well these things will appear.)

One of those things is text-/wordmining. Not only looking at cited references (who cites whom, what texts could be complementary), but also looking concordance and such (how frequently certain words appear in those texts).

Here’s an example:

Clusters of collocate terms in Material Culture coursepack. – Made with Voyant-Tools.org

Also, using all sorts of terms related to material culture, specifically:

Collapsed version of the word trends related to material culture, in coursepack texts for SOCI/ANTH 441 Material Culture. – Made with Voyant-Tools.orgList of terms related to material culture, by frequency. – Made with Voyant-Tools.org

Collapsed version of the word trends related to material culture, in coursepack texts for SOCI/ANTH 441 Material Culture. – Made with Voyant-Tools.orgSeparate trends for words related to material culture, as found in texts for the SOCI/ANTH 441 Material Culture coursepack. – Made with Voyant-Tools.org

Glad you were able to explain the situation in a way your friends could understand. As with many explanation, the important thing is to match it with the context. Many people have a hard time understanding what research entails, even if you describe the whole process (I’m teaching field research, these days). So it’s nice that the match happened.

Thing is, even within academia, some people seem to have misconceptions about sabbaticals, Not that they’re taken to be vacations, but they became so associated with a reward system that they seem not to have their own purpose, anymore.

I’m part-time faculty, so I don’t get sabbaticals in that same sense. And, among the many sources of bitterness among contingent faculty members is the fact that we don’t get the opportunity to apply for sabbaticals. Unlike many of my colleagues, I’m not bitter about my position outside of the tenure-track. And while I understand the importance of sabbaticals, I perceive them as one option among others, to make possible a certain type of academic work. Sabbaticals might even blind some people into misunderstanding their own work.

These are issues with PTR (“Promotion, Tenure, Reappointment”). In that context, the formula used to calculate rewards in the tenure system has become something more of a bureaucratic requirement, and less of a reflection of that work. The percentages themselves are often misleading, as many a professor integrates several of these activities together. When it becomes reality, the dream of cross-fertilization between teaching and research makes percentages more of an accounting exercise than an assessment of the actual work being done. Attempts at making one’s work more efficient often pushes professors to focus their teaching and service work on things which can contribute to their “research output”. And the very categorization of academic work in these three little boxes has been the object of much discussion. Academics typically want to be free to do the work they want to do and are in fact so driven by intrinsic motivation that they’re likely to work more when they’re free to decide how they spend their time.

Something I’ve noticed at institutions where research is the primary thing (for instance, at “Research I” universities in the US) is that not-insignificant number of faculty members have a tendency to publish a rather large number of very similar papers, as if publication were its own goal. At more “comprehensive” institutions (including Concordia, but also a satellite campus in a state university system in the US), it’s not uncommon for faculty members to frequently publish book-length accounts on diverse topics, even though books are often considered as not counting for PTR purposes.
To be clear, I’m not saying that the overwhelming majority of research professors “keep publishing the same article over and over again”. But there is an intriguing pattern in terms of the research focus whereby it may become more about lines in an annual report, whereas research by professors who primarily teach may be more of a “labour of love”.

In other words, while an article can certainly require a lot more work than a book, there has been a tendency for certain research-heavy professors to focus on numerical output instead of on breadth. In the process, the “dissemination” goal of publication may get lost. Not blaming these faculty members, in this. They’re caught in a very constraining system. But the effect remains that some of them may “go through the motions, because they have to” instead of developing the type of research career which is likely to benefit society as a whole.

So, back to sabbaticals… One reason they function in context may be that they allow a sort of “reset”. They bring research back to its roots in deep curiosity and passion for knowledge. Sure, much of it has to do with the time needed to conduct a research project. In field research, it’s obvious that extended field trips are qualitatively different from short stints in the field. In many situations, it is in fact impossible for a scholar to accomplish her work while teaching in even the shortest and least time-consuming of contexts. You simply cannot be in two places at the same time.
But I sincerely think (based on diverse contacts with colleague) that much of the effect felt subjectively by researchers during sabbaticals has more to do with putting things in their proper perspective. During these extended periods of time devoted to their research work, researchers are allowing themselves to explore. While they do have a responsibility to bring back something rather impressive from their time spent away from their more quotidian departmental lives, this pressure is somewhat more indirect, and is less likely to constrain them to “think in the short term”. Chances are relatively high that, during her sabbatical, a given scholar will have an epiphany, will make an life-changing discovery, will go through an actual shift in paradigm, or will cause a sudden leap in scholarship. Any of these things depends on a large amount of work but, mostly, they require a lot of freedom. And freedom is almost the “currency of academia”.

Which brings me back to my own situation outside of the tenure system.
While tenure (and sabbaticals which depend on it) may be associated with academic freedom, there are other ways to be free in academic contexts. One approach which is rarely discussed is to act more as a “free agent” and less as “departmental property”. Belonging to an academic department brings many benefits but it can also be quite constraining. On the opposite end, “contract teaching” can sound extremely constraining but also brings about its own type of freedom. While a tenured professor who goes on sabbatical may accomplish impressive work because she’s free to do so, contingent academic workers are also contributing important work because they are free to work outside of departmental routine. It may well be that, taken on its own, the work of a given adjunct may be much less impressive than that of any full professor. But, in aggregate, contingent academic labour allows for a different type of contribution to knowledge and social change. The same can be said about many non-academics, of course. Several of them could take advantage of a sabbatical, once in a while. In fact, I’ve done a bit of work for a foundation (Carold Institute) which provides for something like a sabbatical for non-academics in the form of a fellowship in support of leaders in community organizations and in diverse spheres of voluntary action.

The fact that sabbaticals are restricted to tenured faculty members isn’t just a matter of rewards and privilege. It’s also about putting research work in its proper context.

[This is truly a placeholder. I did a session at WordCamp Montréal and have been using this post to share slides and other relevant information.]

BuddyPress, the social network based on WordPress, is becoming more and more common in academic environments where it is used for organization, communication and learning. In this discussion we will go beyond the practical and into the human side of things to consider what effects social software might be having in the world of academia. Can we use our tools to get people to think differently? Does the use of open tools help bring openness to the context where they are used? Are BuddyPress and WordPress the best tools for the job? This talk will take the form of a guided discussion including the entire audience.

Audio Recording

(Apologies for the poor sound quality. It was recorded directly from the mixing board used in the PA system, but I wasn’t able to set the recording in the way I should have.)

[Sidenote: I have a problem with journalism (not with journalists, just with their work). As may surprise some people, the NYT appears to me to be part of that problem, not of its solution. And this piece by Trip Gabriel reminds me of diverse things I dislike about journalism. Not that Gabriel’s work is bad or that he’s a bad journalist. But I find in this piece a representation of something problematic with journalism that would take me too long to articulate here. Still, there’s some interesting insight in this piece.]

One thing I find useful in this NYT article is the admission that there might be different approaches to plagiarism and academic integrity. This is something which is discussed among teachers, but it’s the first time I see it in MSM. The very idea that we can expand the conversation about these issues is quite refreshing. Eventually, it might even help us hash out some of these issues in a less visceral and even dispassionate fashion. But I’m not holding my breath on this one.

Another thing I find useful in that “blurred lines” piece is a small set of quotes from fellow ethnographer Susan D. Blum. (Though I was apparently living in South Bend, IN when Dr. Blum became Kellogg fellow for international studies, I don’t recall meeting her.) What Blum says is not only anthropologically relevant but resonates strongly with things I’ve noticed, My summary from these short quotes: values surrounding plagiarism are best understood in a context in which individual ownership of ideas is emphasized; there might be a shift in these values through a much broader cultural change. If I hear Blum correctly, she isn’t taking the usual “techno-determinism” route (“it all happened because of this strange newfangled thing they call the Interwebs”), nor does she assign the causal relationship the other way around (“a shift to a more anonymous culture made it possible for the Internet to expand, in the last twenty years”). Instead, she’s describing a broad phenomenon, based on an understanding of some historical developments of so-called “Western” culture.

Part of the reason I’ve decided to revisit this issue is that the first parts of this series seem to be getting just a bit of traction. As we say in French: «il faut battre le fer pendant qu’il est chaud» (“Strike while the iron is hot.” meaning that we should take advantage of the momentum we get).

Vanessa mentions the COCAL conference, a meeting about “contingent academic labour.” Now, to be honest, while I knew it was happening in Quebec City this summer (during a meeting of Concordia’s part-time faculty union, our president had discussed it), I hadn’t really looked into it. Not that I’m not interested in the issue, but I’m not into formal conferences, anymore.

And (he says sheepishly), I’m worried that my position might not be well-received among (opinion) leaders in this movement. It’s one thing to post my thoughts on these issues and potentially debate them, in writing or face-to-face. It’s quite another to be directly confronted for a significant period of time by what might be a number of vocal activists of what I’ve described as “career part-timers.” I’m all for thoughtful debates and I can stand up and discuss these things publicly. But I do have an issue when things quickly become heated. I may be way off, here, but in my (admittedly limited) experience with groups of “career part-timers,” the heat source has usually not been too far away, when members of these groups meet.

Now, let me stress that I’m not dismissing anything that any kind of part-timer, “career” or otherwise, does. I certainly understand the concerns with precarity and working conditions. I’ve been lucky enough to have been well-represented by strong unions at a number of places where I’ve thought and I do want to involve myself in some activities related to the responsibilities and rights of “contingent academic workers.” I just happen to think that the core issue isn’t so much with part-time work by itself (though I really wish we’d think about alternate models, including long-term contracts). Put bluntly, I’d say that the whole academic system could afford some rethinking. Which could have a huge impact on those who are currently part of the “contingent academic labour force” and those who will be member of the “academic labour” (contingent or not) in the future.

While this long preface has to do with part-timers, I’m really thinking about tenure. Not that tenure should necessarily disappear. But we might as well put some things in perspective. This is what this series is about. Not the defence of one group of people against another (I’d side with learners, if I were allowed to do so). Not a proposal to save academia in its current form. Not even a recipe for building the replacement to the current academia. Just some thoughts about things we may be taking for granted.

So…. I have been thinking about labour issues as I was writing the previous parts in this series. I didn’t mention these labour issues that frequently, but they influence much of what I say about tenure.

“In today’s job world” (at least, what I know of it), academic tenure seems quite unusual a system. Almost anomalous. As mentioned elsewhere, it’s similar to some employment in the public sector, especially in terms of “job security.” But academic tenure is it’s own thing. It’s “kind of special.”

Part of it has to do with prestige. Not only that “university[ref]Partly because, in Quebec, colleges are a separate programme between high school and university, I use “university” for just about any institution of higher education, including some which are called “college.” So, a professor at Bridgewater State College or Tufts College is a “university professor,” in the way I describe things.[/ref] professor” is among the professions with the highest social status in a number of social contexts (between dentist and architect, according to Davis et al. 2007). But there’s a gap between any other kind of academic work (graduate student, senior researcher, etc.) and the prestige associated with a university professor.

More so than a number of other professional groups, the group represented by university professors is often perceived as a select club. At least in the sense that membership doesn’t follow a similar set of guidelines as that of, say, lawyers or engineers. In that sense, other professions have more of a guild system while “academic tenure” is more similar to a club.

(It should be noted that multiple efforts are being made to make this “club” more open to representatives of a number of socially marginalized groups. I don’t mean to say that “the faculty club is still an old boys’ club” since I don’t have data on this. But, even in this case, the process seems quite specific to academic tenure. Sure, complaints about discriminatory hiring may refer to similar rules whether they concern administrators or professors. But it still seems, to me, that the way complaints and claims are discussed shifts, when it’s about academic tenure. )

The contrast between academic tenure and other professional systems seems especially obvious, to me, when thinking about the fact that university professors aren’t specifically trained to be university professors. Oh, sure, they’re trained in their disciplines and may even have some background in pedagogy. Some contexts even have mentoring programes. But there isn’t that clear a path to academic tenure. Many people who want to become university professors end up knowing about the process as they go along, often by talking with other academics. But even some advanced graduate students (including some PhD graduates) seem to be in the dark as to many aspects of academia. They’re expected to learn on their own much of what the job is about (i.e., committee work, advising, grant proposals, etc.). Many high-profile professions have very formal vocational training which makes the kind of preparation new faculty get sound less than minimal.

As a way to protect professionals, tenure also seems quite unusual. Nurses, architects, and surgeons all have orders, professional associations, or other groups to protect members. Such groups are often tied to a national or other state entity of some kind (in Quebec, for instance, these “orders” tend to be provincial). Tenure is given by a specific institution and is tied to appointment in a given “unit” (an academic department, faculty, or programme). There might be rules in terms of synchronizing different institutions’ tenure systems, like there are “transfer credits” and “course equivalencies” for learners. But, if there are, these rules don’t seem to widely known. For, say, surveyors and physicians, it might be very difficult if not impossible to work across national boundaries, but their associations protect them regardless of their place of employment. For university professors, it might actually be very easy to teach in a completely different part of the World but, unless I’m mistaken, there isn’t any “universal” protection going from one institution to the next.

All this to say that “university professor” is quite unlike some other professions.

Part of this may have to do with the fact that academic work isn’t really considered a professional career in the same sense as is dentistry, accounting, or psychiatry. In a way, it’s more like a “calling” like priesthood or a function of a person’s perceived skills, like some conceptions of art. In some people’s minds, the goals of academia are so lofty that, surely, these professional matters are either secondary to “true academic work” or even irrelevant to university professors. Not that faculty members don’t discuss these things at faculty meetings. But there’s a clear separation between “what the work is about” and all matters of professional issues.

Which brings me to unions. In few discussions of tenure is there an explicit notion that what tenure provides is what a union is supposed to provide. Tenure is too unlike union representation to be considered in the same context. Yet much talk about the importance of tenure revolves around things which are, “come to think og it,” pretty much what people in other lines of work consider to be labour issues, managed through unions, management structures, executive boards, and the like.

Part of the potential bias can be attributed to the fact that I grew up in Quebec which is rather union-friendly and had a number of strikes by teachers (as well as by students). The two unions for “contingent academic labour” by which I’ve been represented in Quebec have gone on strike during the past few years (and both have eventually won some key points). In addition, my father spent most of his career as a special education teacher in a high school and what I saw from unions protecting teachers in Quebec was quite impressive to the kid that I was when some key events happened. So I might overestimate the power of unions.

The key notion, though, is that there’s a union-based way to deal with a number of labour issues affecting teachers at many levels, including higher education.

Many (but not all) university professors are unionized. In some cases, these unions are able to offer professors a lot of protection. Probably not as much as the tenure system, but often more than for many other jobs and professions.

As far as I can tell, union protection to university professors is quite effective in dealing with cases of undue firing. There are plenty of edge cases but it actually doesn’t sound like non-tenured but unionized “academic labour” is less protected than most other lines of work. And we’ve probably all heard of cases where tenured professors have been overprotected to the extent that it caused serious problems.

I insist so much on this because the refrain to the song about tenure is “academic freedom.” The notion is that, without tenure, professors wouldn’t have this very special type of freedom that only tenured professors get. Now, I do understand part of the difference between “academic freedom” and more general forms of freedom, including that associated with the set of responsibilities and rights given to people protected by a strong union. But that difference is often left unexplained.

In fact, several people have told me that tenure protects them from being fired from something they might say in class. It’s an interesting issue in its own right but it’s also one which may call for other solutions. Because it’s basically a labour issue. Besides that, though, it raises the question of exactly how free university professors really are. In my experience, people on the tenure-track clearly don’t seem very free at all (it’s almost like an overextended probation period). And, from the outside as well as from direct discussions with “insiders,” even tenured faculty don’t seem that free to work the way they would like.

Which is why I started with “contingent academic labour” (part-time and contract-based university teaching). While, as a context, “contingent academic labour” relates to limited job security (still, much more than that of your typical freelancer), there’s also a very clear freedom involved when you can decide to take or refuse courses, when you’re allowed to work elsewhere at the same time, and where the official “burden” you get is that of building and teaching courses. While it’s not the “carefree” lifestyle of the so-called “Bohemian,” it’s still closer to my personal ideal of freedom than requirements placed on most tenure-track or even a lot of tenured faculty.

So, in summary, tenure seems to be an unusual and probably not that effective a way to deal with professional and labour issues. And it’s probably not that straight a path to genuine freedom.

Got a number of things to say about testbanks, including a sort of “justification” for my use of them and several comments about how I use them. But I thought I’d post a “tidbit” about a specific testbank system. In a way, it’s more “practical” than other things I have to say about testbanks and practical things fit more in the “tidbit” concept I’m using here. I even thought about doing a tutorial. But tutorials are time-consuming and I’m not sure it’d be more useful or useful to more people than this less-structured post.

EZ Test Online (EZTO) is, as you might expect, an online system to facilitate test creation. More specifically, it’s the online version of the EZ Test program developed by McGraw-Hill to distribute the testbanks which accompany its textbooks as “anciliary material.” Almost all of the features of the standalone version of EZ Test are available in EZTO and the overall user experience is quite similar. Haven’t used the offline version in a while but, if I remember correctly, it also used a Web browser (though it could be used without an Internet connection; basically, it was using the local machine as a simplified webhost).

Obvious advantages of the online version over the offline one:

You can work from any machine.

You can share testbanks.

You can have students take tests directly in EZTO.

Obvious disadvantages:

You need an Internet connection.

In some circumstances, it might be slightly less secure than an offline program.

In some ways, you’re sharing data with McGraw-Hill (though their privacy policy doesn’t really specify much about this).

In my case, EZTO makes a good deal of sense. I tend to use “the cloud” as much as possible, these days, and EZTO is convenient in that sense.

In a way, I wish McGraw-Hill were to go further with EZTO. As with Learning Management Systems, I can easily imagine some connection with the “social Web.” Of course, most people would prefer testbank items to remain protected, hidden, and private. But I can still imagine the sharing of testbanks between instructors to become similar to sharing Web content generally.

In fact, I’m slowly coming to terms with a notion of “transparent testbanks.” Basically, it would mean that all potential questions would be available to the wide public, instead of being restricted to instructors. It’s an extreme case, and one which I think most instructors would think is crazy. But I think it might actually make it easier to control such things as cheating, as counterintuitive as it may sound. Even though I’m typically in favour of radical transparency, this is really an edge case and I’m not that attached to it. But knowing that many exam questions are already shared among students, putting all questions in the open (and, one would suspect, making sure that some very similar questions would be part of those testbanks, to prevent mere memorization of the answers) could have actually make this practice of private sharing of questions more impractical.

After all, there are already some sample questions available in diverse places, including on open access websites created by textbook publishers. These questions are typically meant to help students prepare for exams and quizzes. Using the same questions on exams may have the effect of encorcing rote learning. But if there are enough questions in a centralised testbank which serves both as a practice and as a source for exam questions, the result may actually be to avoid using questions which were already seen by individual students.

But, again, I’m not pushing this model. I’m just wondering about the effects. Part of the reason is because of things like information about the “Rorschach test” having been publicised. Haven’t had to deal with this issue directly and what I’ve heard about it was rather minimal, but the fact that test items can now be shared easily means that we need to think about the effects of sharing. After all, even if one is to limit testbank access to instructors of a given course (with all the necessary processes of verification and validation), it only takes one hacked account to make the whole thing insecure. And even the most careful (and even paranoid) of instructors can be the victim of some form of hacking, including “social engineering.” So the assumption that testbank items can be protected clashes with the ease of “leaking content” online. A transparent testbank is a way to flood potential cheaters with an overwhelming number of potential questions, which is similar to disinformation strategies currently used to protect some private information, including trade secrets.

Even in the case of a “transparent testbank,” I imagine much metadata to remain private. In this I include success rates for each question, answers given by individual students, and comments about use of the question. Other pieces of metadata could potentially be shared too, including “source information” about each question (chapter or page number in a textbook), question type, and even aggregate ratings. These can make testbanks look like so much “user-generated content” online. Which could have some advantages.

I didn’t originally plan to make this tidbit about those issues, but I’ve been thinking about them a bit and it’s probably better to put them “out there.”

What I did plan to talk about is how I use EZTO.

So, in some courses I teach, I do use McGraw-Hill textbooks which are accompanied by testbanks in EZ Test format. In fact, I just finished creating an exam using such testbanks.

This is for an online course using Moodle. Students take the actual exam through Moodle. Students have also been taking reading quizzes, with questions from the same testbanks. The midterm exam focuses on understanding. Students have a full week to take the exam (i.e., there’s no set schedule for everyone to take the exam at the same time). In this context, it seems necessary for each student to receive a different set of questions as other students in the class (otherwise, shared questions would guarantee a near-perfect grade for every student who sees them). With individual versions of the exam, risks related to questions being shared are still a matter of concern, but the effects are much less damaging. (In fact, I haven’t noticed any significant effect from questions being shared.) All questions are multiple-choice and they are graded automatically in Moodle.

The key thing, here, is that I want to have a big pool of questions for the exam (to make sure that very little overlap exists between two versions of the exam) yet I want to make sure the questions are as appropriate as possible. The testbanks I use have many questions which are really about rote learning, and that’s not the kind of exam I want to give. I’m also trying to get a nice balance in terms of the types of questions asked. Otherwise, I’d just dump all the testbanks in Moodle and select the appropriate number of questions from the whole set of testbanks.

I’ve used several methods in the past but I think I found the most useful one in this context. There are likely other methods which work well for other people and/or other contexts. But it’s the one I’m likely to use, for the time being.

Here’s what I did… (This is where a tutorial would make sense!)

In EZTO:

Uploaded all the testbanks to EZTO (they’re separated by chapter).

Created new tests for each question category (according to the testbank): definitions, perspectives, etc.

For each test by question category, added questions of that question category from all the chapters. In other words, collected all the questions about definitions from each chapter by using “Select Questions from Banks” with each chapter-based testbank.

In each question category testbank, created a new category to select questions.

Assigned that new category to questions which seemed appropriate for the exam.

Created new testbanks for the selected questions from each question category (again, using the “Select Questions from Banks,” but using the new category to select the questions).

Checked and edited each testbank of selected questions by question category.

Exported each bank of selected questions by question category to Blackboard format.

In Moodle:

Created new question categories and new folders to reflect the exported EZTO testbanks (e.g., a folder and question category for questions about “definitions”).

Uploaded the files exported by EZTO (“bboard.zip”) into the new folders.

Unzipped these Blackboard export files to get multiple files which are all called “res00001.dat” (that’s why it was so important to separate in folders).

Imported questions from each “res00001.dat,” making sure the appropriate category was selected.

Created a “quiz” activity, with a description of the exam, some information about the process, and some settings (can only be taken once, time is limited to 60 minutes, etc.).

Editing the exam, chose each question category and added a number of random questions from that category.

Changed the total on which grades will be calculated (60 points total for 60 questions).

Previewed the exam.

Made the exam visible to students.

Announced the exam.

A fairly convoluted process, but one which provides me with remarkably appropriate results. Actually, other attempts I’ve made, in the past, were even more convoluted. For instance, noticing that several questions revolved around the same concept or issue, I would create subcategories from which only one question was randomly picked. It was very time-consuming and the rule of only having one question per issue or concept was self-imposed. Even with a lot of cautious work, it’s almost impossible to get each version of the exam cover exactly the same topics in exactly the same depth as every other version. Given a sufficient number of questions, one can do as if these things even out. Not ideal. But reasonable in this context.

Selecting questions is neither difficult nor that time-consuming. A lot of questions are easy to eliminate, using a few “rules” I have about exam questions (including “no question about factoids from the text”). There are enough questions in the testbanks I use that I need not worry about having enough for the exam. For this midterm, I had more than 800 questions in the relevant testbanks. I selected a bit less than half of these questions as being appropriate for the exam. The exam contains 60 questions. So, basically, I threw away more than half of the testbank questions and ended up with six times the number of questions for each version of the exam. Unless I’m mistaken, the chances that two students have almost identical versions of the exam are low enough to be insignificant (in a group of 25 students). And, though the questions come from a testbank, they’re appropriate for this specific group.

Much of this could be done through another testbank platform. And, of course, the questions could come from a testbank I’ve created. But, in this case, the quality of the questions I selected is sufficient for me to be rather satisfied. I’ve created other exams through similar methods in similar contexts and the results seemed rather appropriate. If problems arise, it’s still possible to work through issues without losing my sanity.

If I sound like I’m justifying myself, it’s probably because I am. There’s something strange about using testbanks, especially when exam versions are randomised. I wouldn’t use the same strategy in other contexts but this is a course for non-major given purely online. It’s also a course for which the textbook really is the main source. And it so happens that the testbanks are rather decent, containing a good number of questions that I find very appropriate.

The parts which are specific to EZTO mostly have to do with how questions are selected and exported. This is a situation in which I wish I had access to ratings for each question, and some way to bulk process the kind of question I don’t want. (For instance, I exclude all questions which have “All of the above” or “None of the above” as the correct answer.) I’d also want to get some way to assess the overall difficulty of the exam so that I could tweak it up or down depending on what seems to be needed.

I also feel like I’m getting too close to standardized tests. In fact, I keep thinking about the GRE and TOEFL (the two standardized tests I’ve taken). I used to react strongly against them but it sounds as if I were getting closer to them, in this context. Exams in my other courses differ significantly from that model, thanks in part to open questions. And I could technically apply some of the same principles in this case, though it’d make things more difficult for everyone. Yet the reasons I use randomized questions from testbanks have to do with the conditions in which I teach this course, much beyond convenience and time commitment.

In such a context, McGraw-Hill’s EZ Test Online makes some sense. It’s “good enough” as a solution to a given task in a very specific set of conditions.