Friday, July 27, 2012

Something that has always bugged me
is human’s ability to translate subjective, mental information about
probabilities, happiness, guilt etc. into objective, quantifiable numbers. For
me the problem really became pronounced when reading a paper asking students to
assess from 0-100% how likely they felt their answer to a simulated SAT
question was correct. Say that you are given 5 choices: A, B, C, D, and E. Ignore for a
moment the inherent difficulty in forecasting using data, and instead focus on
the process one would go through to make the assessment of how certain they are.
First, you would have to think how well do I know this subject area? If your
answer is ‘not too well’, then you have to translate that ‘not too well’ into
some range. Let’s say I have a 30% chance I know the correct answer with
certainty. This number is compared to the 20% chance that if you guess, you got
it right. However, now you start going through the answers themselves and
determine how ‘reasonable’ they seem. The mental machinations may exclude one
obviously incorrect choice, but now we’re stuck with what do we mean by ‘reasonable’
in the context of some finite number. Let’s say I’m 100% certain D is wrong is wrong
and 85% sure E is wrong. Knowing this means we might as well only select from
A, B and C. With this restricted choice set, my probability of being right,
incorporating my prior belief state, is about 35.29%, now isn’t that a nice
number? However, we are ignoring one key problem, my initial assumption that I
was 30% certain I knew the right answer. How am I to know that because I got
cut off earlier in the day by some bozo, I am now just a little more
pessimistic in my outlook? Because of me being in this “hot
state”, I shave off 10% from my initial assumption. This question gets even more
interesting when looking at how people make absolute comparisons. George
Miller, who recently passed away, was a pioneer in the field of short-term
memory, writing a now rather famous paper entitled, ‘The Magical Number SevenPlus or Minus Two: Some Limits on our Capacity for Processing Information’. One experiment of particular interest to this
discussion deals with peoples’ ability to discern differences in tones, Prof.
Miller sums it up nicely:

“When only
two or three tones were used the listeners never confused them. With four
different tones confusions
were quite rare, but with five or more tones confusions were frequent. With
fourteen different tones the listeners made many mistakes.

These data
are plotted in Fig. 1. Along the bottom is the amount of input information in
bits per stimulus. As the number of alternative tones was increased from 2 to
14, the input information increased from 1 to 3.8 bits. On the ordinate is
plotted the amount of transmitted information. The amount of transmitted information
behaves in much the way we would expect a communication channel to behave; the transmitted
information increases linearly up to about 2 bits and then bends off toward an
asymptote at about 2.5 bits. This value, 2.5 bits, therefore, is what we are
calling the channel capacity of the listener for absolute judgments of pitch.

So now we have
the number 2.5 bits. What does it mean? First, note that 2.5 bits corresponds
to about six equally likely alternatives. The result means that we cannot pick
more than six different pitches that the listener will never confuse. Or,
stated slightly differently, no matter how many alternative tones we ask him to
judge, the best we can expect him to do is to assign them to about six
different classes without error. Or, again, if we know that there were N
alternative stimuli, then his judgment enables us to narrow down the particular
stimulus to one out of N /6.”

The
takeaway from all his results is that humans have an innate capacity to make an
absolute judgment among 7 different items on a uni-dimensional scale. For
example, if someone were given 14 shades of green and asked which ones are
different, humans would usually say that 7 of them are the same and 7 are
different. Now one must remember that these are questions about single
dimensions of OBJECTIVELY knowable items, like color, sound, taste etc. The
world out there is filled with the unknown. When pollsters and academics ask
questions to subjects relating to “enthusiasm to vote”, “dislike with the
president’s economic policy” or “probabilities that your answers are right”,
participants are doing their best to bring all these factors together and spit
out a number. What I’m saying is that the results that these processes glean
may be telling us little about people’s true tastes and instead depend heavily on how
many choices participants are given, as well as, other factors related to
framing of the questions asked.