Modules and Plugins Database - an improved rating system

Details

Description

Just a thought here > I realise we have a rating system for each module & plugin in the database on its respective page (but most ppl don't use it). Moreover there ISN'T a system that actually ranks or compares the credibility of the plugins. We all appreciate the time & effort developers invest to make these plugins available to our community, but lets face it - some are more stable & better maintained than others.

As a point of comparison, Joomla! CMS gives users the opportunity to favour, rate, recommend, & review extensions. Users can also browse by most favoured, top rated, most popular & reviewed.

Do we need to devise a more effective system that makes it easier for Moodle community members to make an informed decision about which plugins are(not) reliable?

Ease of installation, functionality, stability, moodle-like appearance & overall rating are some likely assessment criteria. Administrators, developers & end-users should be able to evaluate the contributions of others. On the same token, this may motivate developers to contribute & further improve their work.

Martin Dougiamas
added a comment - 03/Apr/08 10:47 PM As ideas surface, can there please be some brief analysis with it of
how much is database template configuration and
how much is new code in the database module

use a more meaningful rating scale eg. 1-7 stars as per moodle.org forums. is mapping of existing data possible?

change 'comment' field title to 'review' & use subtext that will focus reviews eg. "having tested this extension, review it based on: ease of installation, functionality, stability, moodle-like appearance etc"

criteria such as ease of installation, functionality, stability, moodle-like appearance & overall rating may be more relevant than 'usefulness'.

New code in database module

count of downloads on a per extension basis will require new code

calculations that display most favoured, recommended, reviewed extensions will require new code

Martin - Here is my summary of what folks are seeking for the Modules and Plugins ratings. First, an ability to specify the role of the person doing the rating (i.e. student, teacher, admin, developer, Moodle Partner?). The characteristics as I summarize/distill them from the discussion at this point are:

One idea I had was to create a feedback for the Modules and Plugins and have the user select the one they are rating. While it would collect the data the feedback module is limited in its ability to filter responses during analysis so that you could see for example how developers rate the quality of code or users rate usability. If we were to do it with feedback, it would be nice if we could limit one response per user per module but feedback does not currently have that capability.

Another idea I had was the possibility of creating another module type that would allow the rating of other activities (modules) with the ability to create the questions similar to feedback. Essentially I'm thinking along the lines of mdl_ratings with a structure of (id, userid, mod, modinstance), mdl_ratings_question (id, questiontype, question) mdl_ratings_questions (id, ratingsid, ratings_question_id, response). This is simply an idea of taking the application from how to rate this particular database activity and abstracting it any activity (assignment, quiz, etc.).

I do not have any ideas about how practical that would be so please know that I'm simply brainstorming. My preference is actually toward the feedback module; however, if were to create a way where by any rating could be customized to include multiple rating scales that could be defined that would also work. The next challenge would be how to display those ratings as it would be ideal to have them graphically visible with the ability to filter according to particular fields (which inclines me back toward feedback). Hopefully some of this makes some sense and might be of use.

Anthony Borrow
added a comment - 13/Jul/08 3:21 AM Martin - Here is my summary of what folks are seeking for the Modules and Plugins ratings. First, an ability to specify the role of the person doing the rating (i.e. student, teacher, admin, developer, Moodle Partner?). The characteristics as I summarize/distill them from the discussion at this point are:
1) Quality of code: 1-10
2) Active maintainer: Y/N
3) Overall rating (0-10) [already implemented as 0-100]
4) Popularity (based on a count of the number of ratings)
5) Review/comments [already implemented as textbox]
6) Documentation: 1-10
7) Usability: 1-10
One idea I had was to create a feedback for the Modules and Plugins and have the user select the one they are rating. While it would collect the data the feedback module is limited in its ability to filter responses during analysis so that you could see for example how developers rate the quality of code or users rate usability. If we were to do it with feedback, it would be nice if we could limit one response per user per module but feedback does not currently have that capability.
Another idea I had was the possibility of creating another module type that would allow the rating of other activities (modules) with the ability to create the questions similar to feedback. Essentially I'm thinking along the lines of mdl_ratings with a structure of (id, userid, mod, modinstance), mdl_ratings_question (id, questiontype, question) mdl_ratings_questions (id, ratingsid, ratings_question_id, response). This is simply an idea of taking the application from how to rate this particular database activity and abstracting it any activity (assignment, quiz, etc.).
I do not have any ideas about how practical that would be so please know that I'm simply brainstorming. My preference is actually toward the feedback module; however, if were to create a way where by any rating could be customized to include multiple rating scales that could be defined that would also work. The next challenge would be how to display those ratings as it would be ideal to have them graphically visible with the ability to filter according to particular fields (which inclines me back toward feedback). Hopefully some of this makes some sense and might be of use.
Peace - Anthony

After giving the issue some thought, and talking to more people about it, rather than spending time customizing the existing modules and plugins database activity, we've come to the conclusion that it would be best to look for an alternative solution. Please see MDLSITE-571. Comments and suggestions are welcome!

Helen Foster
added a comment - 28/Nov/08 1:24 AM Chad and Anthony, many thanks for your comments
After giving the issue some thought, and talking to more people about it, rather than spending time customizing the existing modules and plugins database activity, we've come to the conclusion that it would be best to look for an alternative solution. Please see MDLSITE-571 . Comments and suggestions are welcome!

Just a quick note that I checked with Sun about Netbeans which has a simple but nice looking list of plugins based on popularity and rating. I am beginning to wonder what a good rating algorithm would look like. I tend to think that simplicity is good but we could go anywhere from something like:

to something simpler that factors in the number of ratings and then average of ratings. I'm still searching around for other projects that provide code review. I am thinking of asking in the community to see if we might find some volunteers that would be willing to specialize in very particular things in reviewing the code (i.e. sql injection, data validation, coding style, performance, etc.). If they provided some quality ratings in those areas then we could develop a quality algorithm based on code and some reasonably specific metrics.

I would be interested in ideas from others about what the various rating algorithms could/should look like.

Anthony Borrow
added a comment - 16/Feb/10 11:34 PM Just a quick note that I checked with Sun about Netbeans which has a simple but nice looking list of plugins based on popularity and rating. I am beginning to wonder what a good rating algorithm would look like. I tend to think that simplicity is good but we could go anywhere from something like:
http://www.slideshare.net/spajk/reallife-rating-algorithm
to something simpler that factors in the number of ratings and then average of ratings. I'm still searching around for other projects that provide code review. I am thinking of asking in the community to see if we might find some volunteers that would be willing to specialize in very particular things in reviewing the code (i.e. sql injection, data validation, coding style, performance, etc.). If they provided some quality ratings in those areas then we could develop a quality algorithm based on code and some reasonably specific metrics.
I would be interested in ideas from others about what the various rating algorithms could/should look like.
Peace - Anthony

gavin henrick
added a comment - 17/Feb/10 12:00 AM I think there is a range of issues to be assessed, i currently assess modules based on a few issues some of these are below and I am sure others will have even more they use:
Downloading/Installing
Availability on trusted download site
Ease of install
Documentation for install
Type of addon (patching core files, or a block etc..)
Project
Will it be there in 1 year, is it available for all versions, or some
Project Maintainability (Is it one guy who made something once and isnt around anymore)
Project Version support (only latest moodle? or all moodles since 1.6 ?)
Project life (new project or old)
Project Backer (Paid for ongoing updates..)
Code Design (extra Dependencies/php versions/etc table indexing)
Popularity (recommended/lots of downloads)
Impact on Existing Moodle
Security (XSS, etc..)
Performance (heavy queries, complex code... slow down pages?)
Resource usage (take a lot of disk space, or db or none ...)
backup/restore works?
unit tests?
Actual Usage
Documentation for usage
Interface usability - teacher/admin
Interface usability - student
Is it what it says it is ?
These are some of the things people need to think about before installing an addon, and maybe others?

@Gavin: thanks for your comment - I agree with your ideas. You provide some key considerations relating to the evaluation of third party plugins. Do you have any thoughts in terms of how these consideration might best be represented on a web form, page - so users can provide evaluative feedback of benefit to other moodlers who are investigating the merit of third party plugins?

Chad Outten
added a comment - 17/Feb/10 3:47 PM @Gavin: thanks for your comment - I agree with your ideas. You provide some key considerations relating to the evaluation of third party plugins. Do you have any thoughts in terms of how these consideration might best be represented on a web form, page - so users can provide evaluative feedback of benefit to other moodlers who are investigating the merit of third party plugins?

my 2 cents of quick contribution (sorry, my spare time is really nothing).
Some automatic indicators of plugin quality shoud be:
-> the fame of the author/s (age of his/her first moodle login, number of posts, number of bug fixed, number of posts rated as useful, if he/she is a developer, (some ideas could be fetched from ohloh)...)
-> number of download per time unit of the plugin
-> date of last update
-> number of updates per week/months/semester
-> rate from users using the plugin

Daniele Cordella
added a comment - 17/Feb/10 8:51 PM my 2 cents of quick contribution (sorry, my spare time is really nothing).
Some automatic indicators of plugin quality shoud be:
-> the fame of the author/s (age of his/her first moodle login, number of posts, number of bug fixed, number of posts rated as useful, if he/she is a developer, (some ideas could be fetched from ohloh)...)
-> number of download per time unit of the plugin
-> date of last update
-> number of updates per week/months/semester
-> rate from users using the plugin
Ciao.

Related to this, I have at least one plugin that is rated '0/100' on the back of only 1 person rating it (with no feedback given as to exactly why they gave it such a rating).

I'd be disappointed if such a rating immediately buried the module, never to be seen again (particularly as it's a plugin that I find very useful, as do quite a lot of other people who have emailed me or posted on forums about it).

Davo Smith
added a comment - 27/Oct/10 2:35 AM Related to this, I have at least one plugin that is rated '0/100' on the back of only 1 person rating it (with no feedback given as to exactly why they gave it such a rating).
I'd be disappointed if such a rating immediately buried the module, never to be seen again (particularly as it's a plugin that I find very useful, as do quite a lot of other people who have emailed me or posted on forums about it).