I am trying to assign scores to phone's specifications to calculate a final rating score for each phone.

However, when I was scoring based on megapixels, I found that one of the phones 'A' has a 41MP camera. But all other phones have 8MP or less, which is pretty good for a phone. Right now, when I grade relatively wrt phone A (rating 5/5), all other phones get a score of 1 or 2 out of 5. I would like it to be a little more fair, as it was just one phone with a very high value. Could you suggest me the best way to go about solving this issue where the relative grading is more fair?

Are you aware that "number of megapixels" is an absolutely terrible metric for use in assessing digital cameras of any type, but particularly phone cameras where the sensors are all tiny?
–
jozzasAug 27 '12 at 6:57

1

To be more constructive, maybe try to find a site that has real-world image quality tests for each of the cameras. Assigning points to a megapixel value is worse than meaningless, as it skews your results in an irrelevant way.
–
jozzasAug 27 '12 at 6:58

@jozzas I understand that megapixels is definitely not a good metric for the camera's quality. I will be using megapixels as only one of the many parameters while assigning scores. I am trying to find a solution to a more general problem regarding how to avoid data from being skewed just because of one/few high scores. This might apply to school grades or rating product specs etc?
–
vasaAug 27 '12 at 7:08

4

@user38814: using a meaningless value as "one of many" doesn't really improve things. Many meaningless values added together don't in general produce a meaningful value.
–
Joachim SauerAug 27 '12 at 8:48

Could you please elaborate a little more on it. As in, which scores are usually considered for log? Why would it not be linear?. Please be gentle: I'm no statistician or mathematician.
–
vasaAug 27 '12 at 7:29

Scores are what you make of them. log(n) is just one method to where metrics go up exponentially to flatten them out. The difference between 8 and 41 is quite large and lens itself for this method. After, the score look like: 6.9 vs 7.6. It's just scores and highly arbitrary. But use methods you're comfortable with.
–
Pieter BAug 27 '12 at 7:36

This is called "grading on a curve", and reams of opinion have been written on the pros and cons of that idea. The question whether it is more "fair" to give scores that reflect the proportions of the values you measure, or to give scores that always contain a set number of high and low scores, is probably philosophically unanswerable. Whatever you do, make sure that the users actually know what your ratings mean. If a low rating only means "worst in the review", say so. If it means "approximately half as good as product Y", say so as well. (As others have pointed out, pixel counting is also a bad measure to estimate the quality of mobile cameras. But deciding what values to measure is a whole different issue, also very contentious.)

Despite the fact "megapixels" is a bad metric - you may define some explicit intervals for this kind of thing. For example, you assign these values

1P : 2 MP

2P : 3 MP

3P : 5 MP

4P : 8 MP

5P : 15 MP

For values above 15MP, you give 5 points, for other values you interpolate linearly. For example: 12 MP will result in

(12-8)/(15-8) + 4P = 4.428... points

It may be a bit more complicated to implement than @PieterB's suggestion, but the advantage is that you can adapt it easily to whatever scoring you "feel" to be fair. There's is nothing more to be expected due to the arbitrary nature of such a scoring.

Note that your scoring schematics has to be adapted for every dimension, for example, if you are giving a score for "battery time", 41 hours to 8 hours would most probably need a different scoring. And for "monthly costs", 41 euros/dollars/yen compared to 8 units of the same currency may need a reverse scoring. So whatever you do, there is no "one size fits all" solution.