(f ’ E(F))

E(P|F = f) = E(P) +

σF

2

18.2.9 Comparing a Predicted Field and its

= a + bf (18.13) Predictand. So far we have considered the

γ2 prediction of a single number. Evaluation of such

Var(P|F = f) = σ P ’ 2 .

2

(18.14) a forecast requires many samples in order to

σF

estimate the skill scores in [18.2.3]. When we

The conditional expectation consists of a constant have a vector or ¬eld of forecasts, the skill of a

term a and a term bf that is linear in f. The given forecast can be estimated using scores such

as the anomaly correlation coef¬cient ρ F P (t) or

A

conditional variance is independent of f.6

The moments of the forecast F conditional on the mean squared error S F P (t) which measure

2

the observation p may be derived. the similarity of two ¬elds relative to a given

climatology C.

Suppose f„ (t) is a forecast of a ¬eld, say

18.2.8 Improvement of a Quantitative Forecast.

Any forecast F can be improved statistically if Southern Hemisphere 500 mb height, for the time

we have access to a large sample of previous t prepared „ days in advance, and suppose that the

forecasts ft and corresponding predictands pt . predictand p(t) is the analysis of that ¬eld on the

This improvement can be obtained by regressing day t. Let C be the observed long-term mean ¬eld.

the predictand on the forecast using a simple Then the anomaly ¬elds,

f„ (t) = f„ (t) ’ C and p (t) = p(t) ’ C,

linear regression model of the form suggested by

equation (18.13). Least squares estimators of a

are compared using the anomaly correlation,

bias correction a and an amplitude correction b

(f„ ’ f„ )(p ’ p )

are obtained in the familiar way. We will assume

ρ F P (t) = ,

A

(18.17)

that the forecasts are already unbiased, so that a

(f„ ’ f„ )2 (p ’ p )2

is approximately zero, even though bias correction

does not affect the correlation skill score. where the notation · denotes an area weighted

The outcome of the exercise might be an mean. The time argument (t) has been suppressed

amplitude correction b = 1 so that an on the right hand side of (18.17) for convenience.

improved forecast, say F, is given by F = The mean squared error is computed similarly

F. In this case nothing is gained. The tools as

of Chapter 8 show us that for large samples

(p (t) ’ f„ (t))2

(S F P )2 (t) = .

A

(18.18)

6 See also Section 8.2.

1

18.3: The Murphy“Epstein Decomposition 399

The quantity in the denominator is the sum of the

area weights.

Note that both the anomaly correlation coef¬-

cient and the mean squared error are de¬ned for

an individual forecast. Therefore, an annual cycle

of these scores can be calculated, and the grad-

ual improvement of weather forecast models can

be monitored by these measures. An interesting

aspect of these scores is that forecasts can be

strati¬ed by their success. Thus it may be possible

to understand empirically why some forecasts are

more successful than others.

18.2.10 Example: US NMC Weather Fore-

casts. Branstator [61] and Kalnay, Kanamitsu,

and Baker [209] analysed the quality of the op-

erational forecasts prepared by the US National

Meteorological Center (US NMC). Both consid-

ered the Northern Hemisphere 500 mb height ¬eld,

and monitored the forecast performance using the

anomaly correlation coef¬cients ρ F P .

A

Branstator evaluated three-day forecasts for 11

winters (de¬ned as November to March“NDJFM)

from 1974 to 1985. Time series of ρ F P are shown

A

in Figure 18.4 for three winters, 1974/75, 1978/79,

and 1982/83. We see that the anomaly skill

score of the forecasts gradually improved, from

approximately 0.65 in 1974/75, to approximately

0.75 in 1978/79 and 0.80 in the winter 1982/83.

However, the anomaly correlation skill score

shows a remarkable variability within a winter.

There are periods (e.g., mid January 1978 to

mid February 1978) when the forecast scores

Figure 18.4: Daily time series of the anomaly

are consistently better than during other periods

correlation coef¬cient ρ F P of three-day forecasts

A

(e.g., after mid February 1978). It is not clear

of Northern Hemisphere 500 mb height ¬eld

if these variations tell us something about the

prepared by the US National Meteorological

numerical weather prediction model (i.e., that the

Center during the winters of 1974/75, 1978/79 and

model scores better with certain initial states than

1982/83. Winter is de¬ned as the November to

with others), or if they tell us something about

March cold season. From Branstator [61].

variations in the predictability of the atmospheric

circulation.7

The distribution of ρ F P , shown in Figure 18.5

A line (see [18.3.5]) for about 4.5 days in the early

1980s. By the end of the decade, the skill curves

for the ¬rst six winters and the last ¬ve winters,

stayed above this skill threshold for about 7 days.

is clearly not normal. The skill varies between

Figure 18.6 is a representation of forecast skill

0.5 and 0.9 during ¬rst six winters, and it varies

which is typically used by operational weather

between 0.65 and 0.95 during the last ¬ve winters.

forecast centres to document their progress. We

Kalnay et al. [209] calculated the DJF seasonal

return to this example in [18.4.5].

mean anomaly correlation coef¬cient for lags from

1 to 10 days (Figure 18.6) for the period 1981/82

to 1989/90. The curves lie above the magical 60%

18.3 The Murphy“Epstein

7 Branstator [61] performed a spectral analysis of the skill

Decomposition

score time series and found a red power spectrum, similar

to that of the (predicted) height ¬eld. He suggested that this

18.3.1 Introduction. In this Section we intro-

similarity could indicate that the swings in the skill score re¬‚ect

duce the Murphy“Epstein decomposition of the

the varying predictability of the atmosphere.

18: Forecast Quality Evaluation

400

Figure 18.5: Frequency distributions of the

Figure 18.6: Seasonal mean anomaly correlation

anomaly correlation coef¬cient ρ F P of three-day

A

coef¬cients ρ F P for 1- to 10-day lead forecasts

A

forecasts of Northern Hemisphere 500 mb height

of the Northern Hemisphere 500 mb height ¬eld

¬eld prepared by the US National Meteorological

prepared by the US National Meteorological