The applications of climate models are very diverse in scale and parameters, and “a succinct set of measures that assess what is important to climate has yet to be identified.” A second reason is that the opportunities to test climate model skills are limited because of the slow evolution of climatologies. Furthermore, data are not error free and uncertainties are often poorly underestimated. Gleckler et al. also mention that models can be tuned to appear realistic for some features, “but as a result of compensating errors.” They go on to tackle very elegantly and thoroughly the complex issue of climate model verification offering a methodology to condense the relevant information content as thoroughly as possible without oversimplification.

Both the methodology and the conclusions of the paper are very important and the authors discuss them in the wider context of the complexities of climate modeling, which are laid out clearly along with their significance for the model performance. By use of different datasets, and a variety of indices of model performance, Gleckler et al. rank several models based on their difference with observations. They also introduce two indices to evaluate the mean fields and their variability. Their main conclusions can briefly be summarized as follows:

• The mean and median model very often perform best.
• Generally speaking there are models which tend to perform better than others, but a single model can change its ranking by 6 or 7 slots depending on the field analyzed, whether the focus is on its mean or variance, on the geographical area (Northern Hemisphere, Tropics, etc), and also depending on which dataset is used as reference.
• The mean climate metrics can be “woefully inadequate for describing the multiple facets of model performance”, and a good agreement with observation for the mean climate does not necessarily imply a good representation of the climate variability.

Gleckler, Taylor, and Doutriaux deserve credit for this very important work, especially because they were able to look at model performance in-depth, summarizing the wide variety of information that model output provides. Furthermore, the first two points mentioned above represent a formalization of some aspects of modeling in general that have been implicitly accepted for a while, at least in the NWP community. Because no model can yet encompass alone, the variety of processes and interactions that occur in the atmosphere, no model outperforms all other models in all cases, and a multimodel ensemble can capture additional uncertainties, and thus its mean performs best (for a more in-depth discussion see Hagedorn et al. 2005). The third point raises another important issue: what conclusions can be drawn from a model simulation that does reproduce a mean variable, but fails to characterize its variability? Does this imply that the model is getting the mean value right for the wrong reasons? Most likely so, but we all agree that the significance of the error varies with the variable and the type of mean, and most of all this does not imply that the entire simulation is to be discarded for all applications nor that it can’t be used as a forecast or as a sensitivity experiment. However there is no quantitative analysis, at least to my knowledge, that tackles this issue in a general fashion, although multimodel ensembles are definitely a way of doing just that. While the significance of this problem for NWP is strongly limited by the constant verification of model performance, I think it is an issue for climate-type simulations especially when they are used to establish policies. How significant is a hydrological balance for the future European climate as provided by a global model that does not well simulate the North Atlantic Oscillation? If the NAO is not well captured, the storm track might not be realistic, further decreasing the confidence on the precipitation fields over Europe. Being able to quantitatively assess this type of uncertainty is a very important step toward a realistic use of models.

Reference:
Hagedorn, R., F.J. Doblas-Reyes, and T.N. Palmer, 2005: The rationale behind the success of multi-model ensembles in seasonal forecasting - I. Basic concept. Tellus Series A, 57:219-233.    Source