Developing a Rubric for OLM on Rails

Hey all – I’m Mike, and I’ve just started working on the OLM on Rails project. My first task is to develop the Rubric models, controllers, and user experience, and I’m going to share some musings on it.

Ok, so first off, here’s the idea: A professor creates an Assignment, and has to have a marking scheme to go along with it. The Rubric that she develops will be used by the TA’s to mark the assignments. Sounds pretty simple.

I’m still getting my thinking wrapped together here, so I’m just going to muse for a bit. Here are some thoughts, rules, and caveats:

Creation of a Rubric is a separate event from creating an Assignment. So if an Assignment exists, it does not imply that an associated Rubric exists yet

A Rubric consists of an arbitrary number of criteria to grade the student

Each criterion has a weight assigned to it (ie: 0.25, 0.5, etc)

Each criterion has a number of levels that can be achieved by a student. By default, these levels are 0, 1, 2, 3, and 4 – with 0 being the worst performance, and 4 being the best. We’re considering having the ability to add additional levels, to improve flexibility

An assignment cannot be marked until a Rubric has been created

A Rubric is not valid unless all of the weights of its criteria sum to 1.0 (100%)

The total number of marks achievable on an Assignment equals the sum of each criterion weight, multiplied by the maximum level of that criterion. This does not include possible bonus marks, which are not handled by the Rubric

Assigning weights to criteria is going to be a little tricky, interface-wise. While it’s tempting to just use a simple text-box, and trust the user to input the weights manually, it’d be nice for something a little more elegant and user friendly.

Some professors may want a simpler way of inputting this information quickly, for instance, by uploading a file describing what they want. Andrew has suggested that users upload a YAML file… something worth thinking about.

That’s all I’ve really got rolling in my head about it right now.

So, what have I done? I’ve created two new models: RubricCriteria and RubricLevel. A RubricCriteria is assigned to a single Assignment. A RubricLevel is assigned to a single RubricCriteria. Piece of cake.

I’ve also created some basic unit tests for these models. I think I’m going to start thinking about user interfaces now, because I think it will give me some hints on what I’ll want the Controller to do. I might upload some of my sketches later.

2 Responses

Why not have a single max_level field instead of another Model? That seems simpler to me.

I’m assuming that Rubrics are separate from Assignments so that Rubrics can be reused. Is that correct? And if I adjust the weights on Assignment3, does that affect the weights on Assignment4 (which uses the same Rubric)?

What if the prof wants to set the max_grade_possible parameter themselves, without fiddling with criteria and weights? I’d expect that to be common; my profs know in advance how much they want an assignment to be worth, and they’ll have recalculation woes whenever a new criterion is added.

Why are weights tricky? If you let the prof set the max_grade_possible parameter, they can set weights however they like and you can normalize the scores on the backend.

During one of the preliminary architecture meetings, we decided that we were going to try to keep this system as flexible as possible, but also to provide nice defaults.

By default, every new Criterion will start off with a set of 4 levels – this is true in most CS marking schemes, according to Prof. Reid. Having the RubricLevel model certainly allows us to have those 4 levels, but it also allows us to have an arbitrary number of levels per criterion should the need arise. So, while providing the functionality that we require, I’ve also kept the door open for expansion, or adaptation in the event of a weirder marking scheme.

I’m not sure what you mean by ‘Rubrics are separate from Assignments’ – they’re separate because they’re not the same thing. As far as I know, each Assignment will only have 1 Rubric, consisting of an arbitrary number of criteria, with a default of 4 levels per criterion, and the resulting rubric *will not be reused*.

Weights themselves are trivial. It’s the user interface to set the weights that’s going to be a little tricky. It’s easy to just slap on a text field and trust the user to input a number between 0.0 and 1.0 (or 0 and 100), but that’s horrible UI design. I’ve come up with something a little bit nicer, and I’ll post details soon!

I really appreciate the comments though! If I’m not seeing something, and I could reduce the complexity of these models *without compromising flexibility*, I’m all ears! 🙂