4
How good is a forecast? Overview of talk –Which skill scores have the most desirable properties? –How does skill depend on spatial scale, lead time etc? –If it has an inverse-exponential decay with forecast lead time, what is the half-life of the forecast? –Most model comparisons evaluate the cloud climatology –What about individual forecasts? –Standard measure shows forecast half- life of ~8 days (left) –But virtually insensitive to clouds! ECMWF 500-hPa geopotential anomaly correlation

7
Desirable properties of skill scores Equitable: all random forecasts score zero –This is essential! –Note that forecasting the right climatology versus height but with no other skill should also score zero Proper: not possible to hedge your bets –Some scores reward under- or over-prediction (e.g. hit rate) –Jolliffe and Stephenson: not possible to be equitable and proper! Independence of how often cloud occurs –Almost all scores asymptote to 0 or 1 for vanishingly rare events Dependence on 10x10 joint PDF, not just 2x2 table –Difference between cloud fraction of 0.9 and 1 is as important for radiation as a difference between 0 and 0.1 Linearity: so that can fit an inverse exponential –Some scores (Yules Q) saturate at the high-skill end

12
Summary Half-life of a cloud forecast is between 2.5 and 5 days –Relatively insensitive to skill score (provided a good one is used) –Compare to ~8 days for ECMWF 500-hPa geopotential height forecast –Skill at forecasting cloud increases somewhat for larger scale features Important to assess the merits of various skill scores –At least 5 criteria to judge against, and none are good on all –Plenty of bad ones to use (hit rate, false-alarm rate etc)! –Worth trying Stephensonss Extreme Dependency Score, which is good for very rare events Wish list –Obtain Met Office cloud forecasts beyond a lead time of 3 days –Compare skill of the Met Office model at different model resolutions, but averaged to the same scale –Can we see what skill comes from global model at boundaries, what comes from mesoscale data assimilation etc?

19
Monthly skill versus time Measure of the skill of forecasting cloud fraction>0.05 –Comparing models using similar forecast lead time –Compared with the persistence forecast (yesterdays measurements) Lower skill in summer convective events

20
Skill versus lead time Unsurprisingly UK model most accurate in UK, German model most accurate in Germany! Half-life of cloud forecast ~2 days More challenging test than 500- hPa geopotential (half-life ~8 days)