Comments on Judging predictions of rainfallTypePad2010-06-04T04:58:14Zjunkchartshttp://junkcharts.typepad.com/numbersruleyourworld/tag:typepad.com,2003:http://junkcharts.typepad.com/numbersruleyourworld/2010/06/judging-predictions-of-rainfall/comments/atom.xml/Matthew Fallshaw commented on 'Judging predictions of rainfall'tag:typepad.com,2003:6a00d8341e992c53ef0133f038bb05970b2010-06-06T06:51:32Z2010-06-06T06:51:32ZMatthew Fallshawhttp://tricycledevelopments.com/projects/it-projects/projects/and-a-handfull-of-others/Yudkowsky explains scoring rules pretty well at http://yudkowsky.net/rational/technical#probability-density. See http://predictionbook.com/ for the only publicly available free app I know of...<p>Yudkowsky explains scoring rules pretty well at <a href="http://yudkowsky.net/rational/technical#probability-density." rel="nofollow">http://yudkowsky.net/rational/technical#probability-density.</a></p>
<p>See <a href="http://predictionbook.com/" rel="nofollow">http://predictionbook.com/</a> for the only publicly available free app I know of that&#39;ll help you keep track of your calibration as you suggest (disclaimer: personal interest in PredictionBook). </p>Åse commented on 'Judging predictions of rainfall'tag:typepad.com,2003:6a00d8341e992c53ef013483628a82970c2010-06-06T06:46:06Z2010-06-06T06:46:06ZÅse10 or so years ago when I took judgment and decision making, we discussed the relationship between accuracy and confidence...<p>10 or so years ago when I took judgment and decision making, we discussed the relationship between accuracy and confidence and, indeed, that type of measure was brought up to test that relationship. And I distinctly remembered that meteorologists were actually very good at this.</p>
<p>This type of measure is used when you research meta-cognition. So, for example, when we research eyewitness memory here at Lund we ask people to first recall what happened in a film, and then they note how confident they are in their memory. And, as part of the analysis of meta-cognition (how good you are at judging how good your memories are), we come up with that kind of chart - and a number of additional measures. </p>
<p>This has also been done on metacognition about your own performance (main finding - those who perform the worst tend to be bad at judging how bad they are). </p>
<p><a href="http://www.informaworld.com/smpp/content~content=a905567536&db=all" rel="nofollow">http://www.informaworld.com/smpp/content~content=a905567536&amp;db=all</a></p>
<p>That&#39;s a link to our paper (which I would guess is behind a university library wall, for those that do not have university access.</p>Bill commented on 'Judging predictions of rainfall'tag:typepad.com,2003:6a00d8341e992c53ef01348346beed970c2010-06-04T21:25:20Z2010-06-04T21:25:20ZBillHave you look at "proper scoring rules"? They might be just what you are looking for. For example, each day...<p>Have you look at &quot;proper scoring rules&quot;? They might be just what you are looking for. </p>
<p>For example, each day the forecaster says that the probability of rain is p. Then each day give a score of 1+ln p if it rains and a score of 1+ln(1-p) if it doesn&#39;t.</p>
<p>This rule gives the right incentives to the forecasters to choose a p that best represents their knowledge. I believe that some weather forecasters are scored based on proper scoring rules such as this one. </p>gary commented on 'Judging predictions of rainfall'tag:typepad.com,2003:6a00d8341e992c53ef0133f00b7118970b2010-06-04T19:04:01Z2010-06-04T19:04:01ZgaryI remember when I was young my father telling me that a 30% chance of rain meant that at any...<p>I remember when I was young my father telling me that a 30% chance of rain meant that at any given time, it would be raining over 30% of the forecast area. I asked him, wouldn&#39;t that be 100% chance of rain?</p>floormaster squeeze commented on 'Judging predictions of rainfall'tag:typepad.com,2003:6a00d8341e992c53ef0133f00162aa970b2010-06-04T16:53:50Z2010-06-04T16:53:50Zfloormaster squeezeI always thought probability also included an element of spatial probability (even if there is rain in an "area" it...<p>I always thought probability also included an element of spatial probability (even if there is rain in an &quot;area&quot; it may not hit the whole area) and Nick&#39;s NOAA link confirms that (although it does convey a lot of things that I had not considered). Any validation would involve a very complex physical mapping of rain &quot;coverage&quot;.</p>Nick commented on 'Judging predictions of rainfall'tag:typepad.com,2003:6a00d8341e992c53ef0134832a3f50970c2010-06-04T14:54:59Z2010-06-04T14:54:59ZNickI managed to dig up an article that I remembered running across a while ago that discusses this: http://www.nssl.noaa.gov/users/brooks/public_html/prob/Probability.html Ideally,...<p>I managed to dig up an article that I remembered running across a while ago that discusses this:<br />
<a href="http://www.nssl.noaa.gov/users/brooks/public_html/prob/Probability.html" rel="nofollow">http://www.nssl.noaa.gov/users/brooks/public_html/prob/Probability.html</a><br />
Ideally, a validation system would do a few things simultaneously; however, I&#39;m not sure it&#39;s actually possible to do all of them at the same time.<br />
1) Make sure that, 10% of the time that a 10% forecast is given, it rains.<br />
2) &quot;Punish&quot; the forecaster for playing the system - for example, if the 10% forecasts have been averaging 8%, giving a 10% forecast on a couple of days it&#39;s extremely likely to rain to balance it out.<br />
3) &quot;Reward&quot; the forecaster for making their forecast as different as possible from the past average for that time period. If all the days in July over the past 50 years have had rain on 15% of the days, making a 15% forecast provides no additional knowledge.<br />
</p>Brad commented on 'Judging predictions of rainfall'tag:typepad.com,2003:6a00d8341e992c53ef0133f0004b0e970b2010-06-04T12:29:51Z2010-06-04T12:29:51ZBradI've always been amused at weather sites like wunderground.com and weather.com that will claim a 10% chance of rain on...<p>I&#39;ve always been amused at weather sites like wunderground.com and weather.com that will claim a 10% chance of rain on a given day, but then you click the link for the hour-by-hour forecast, and it lists a 10% chance of rain at 9AM, 10% chance at 10AM, 10% chance at 11AM... If all of these probabilities were independent (which I realize they aren&#39;t), then a 10% chance of rain every hour for 24 hours would lead to around a 92% chance of rain on that day. I highly doubt these hourly forecasts would stand up well to validation.</p>