On the accuracy of short-term COVID-19 fatality forecasts

We collected data on CDC ensemble forecasts between June 2020 and June 2021 [1]. Ensemble forecasts are available for cumulative and weekly incidence numbers and a forecasting horizon between one to four weeks. All forecasts use data from the Johns Hopkins Coronavirus Resource Center [8] as reference. Forecasts are made for epidemiological weeks which run Sunday through Saturday. As an example, if forecasts with one and four-week forecasting horizons are being made on June 7, 2020 the corresponding target dates are June 13, 2020 and July 4, 2020 [9].

We compare CDC ensemble forecasts of COVID-19 fatalities with a simple and easily interpretable forecasting method. To do so, let $y(t)$ be the incidence of COVID-19 fatalities at time $t$. We use $\dot{y}(t)$ to denote the rate of change of $y(t)$ at time $t$. Forecasting the incidence $y(t+\Delta t)$ at a target time $t+\Delta t$ requires us to find an estimate of this quantity at an earlier time $t$. A straightforward way to construct short-term forecasts is to use the current rate of change $\dot{y}(t)$ and determine a forecast at time $t_{k}=t_{0}+k\Delta t$ according to the Euler method [6, 7]

where $\Delta t$ and $k=1,2,\dots$ represent a time step (e.g., one week) and the number of time steps in the forecasting horizon, respectively. However, observed incidences are subject to observation noise that results from confounding factors including sampling bias, measurement errors, and reporting delays [10].

A possible way to “de-noise” observed data is to use previous weekly incidences instead of daily incidence levels. If observational noise can be reduced by averaging over a period of several days, daily errors are less pronounced on a weekly level. However, the local daily derivative is quite sensitive to noise and our incidence correction term is not helping in making accurate short-term forecasts. Therefore, we can impose some degree of regularity to reduce the level of noise with the following minimization

where $y_{k}=y(t_{0}+k\Delta t)$, $w_{k}=w(t_{0}+k\Delta t)$ is a regularized approximation of $y_{k}$, and $\lambda$ is a regularization parameter. In the limit $\lambda\rightarrow 0$, the argument of Eq. (2) is minimized if $w(t)$ approaches $y(t)$. In the limit $\lambda\rightarrow\infty$, the argument of Eq. (2) is minimized if $w(t)$ is constant (i.e., if $w_{k}-w_{k-1}=0$). This optimization process has its equivalent Euler–Lagrange formulation for differentiation [11, 12]. Values of $\lambda\in(0,\infty)$ yield functions $w(t)$ that are smoothened versions of $y(t)$ with respect to the discrete rate of change $w_{k}-w_{k-1}$. Finally, the regularized Euler short-term forecast¹¹1All optimization procedures in this work are applied in a causal manner. That is, at the prediction time $T$ only historical data $y(t\leq T)$ is being used in the minimization (2). is given by

In the following section, we utilize the regularized Euler method to generate forecasts of reported COVID-19 fatalities.

Our source codes are publicly available at [13].

Figure 1 shows CDC ensemble forecasts (solid blue lines) of the weekly incidences of reported COVID-19 fatalities. The dashed black lines indicate reported COVID-19 fatalities. Between June and early November 2020, the majority of reported cases were close to the ensemble forecast. As COVID-19 deaths surged in November 2020, the forecasts of the ensemble method became less accurate than in previous months.

For a comparison between the CDC ensemble point estimates and those obtained with the regularized Euler method [Eq. (3)], Fig. 1 also shows Euler-method forecasts (solid red lines) of weekly incidences of COVID-19 fatalities in the U.S. We observe that one-week CDC ensemble forecast for the majority of data points are not more accurate than one-week Euler forecasts [Fig. 1(a)], which we use as a local-derivative-based forecasting benchmark. Although Euler and CDC forecasts still exhibit a similar structure for a four-week forecasting horizon [Fig. 1(b)], the Euler method is associated with larger deviations from the reported fatalities than the CDC ensemble method. To quantify differences in forecasting errors between the two methods, we use

to denote the absolute error between the Euler forecast $x(t)$ and CDC forecast $y(t)$ for target time $t$.

Figure 1(c,d) show the 4-week moving averages of weekly forecasting errors $\delta(t)$ (solid lines) of the Euler (red) and CDC ensemble (blue) methods. As suggested by our above discussion of Fig. 1(a,b), we observe that the error of the Euler method is substantially smaller than that of the ensemble forecast for a one-week forecasting horizon. In about 61% of the forecasting instances shown in Fig. 1(a), the regularized Euler method has a smaller error than the CDC ensemble forecast. The cumulative forecasting errors are 49,925 (Euler) and 52,885 (CDC). Without correction term [i.e., for $k=0$ in Eq. (3)], the cumulative forecasting error of the Euler method is 52,660, again smaller than that of CDC ensemble forecast. Note that no regularization corresponds to a simple shift of the incidence curve. For a 4-week forecasting horizon [Fig. 1(d)], the cumulative error of the CDC ensemble forecast is 87,717, about 35% smaller than that of the Euler method.

Our results suggest that easily interpretable methods like the Euler method, a model-free local-derivative-based forecasting benchmark, provide an effective alternative to more complex epidemic forecasting frameworks on short-term forecasting horizons. Similar conclusions were drawn in a recent study [14] that compared Euler-like forecasts with those generated by Google Flu Trends. Regularized Euler forecasts have smaller errors with respect to CDC ensemble forecasts on one-week forecasting horizons in about 61% of all cases. Simple curve shifts without regularization provide better one-week forecasts in 63% of all cases, yet with a mean absolute error that is about 5% larger than that found for regularized Euler forecasts. For longer forecasting horizons, it is not surprising that CDC forecasts that rely on additional input data, and epidemiological and statistical models become more accurate than Euler-like forecasting benchmarks.

One clear advantage of Euler forecasting methods is that they are less labor and resource intensive than more complex forecasting models, which often rely on the knowledge of expert groups and require specialized computing infrastructure. In their simplest implementation, Euler forecasts use the currently observed incidence rate as an estimate of the incidence rate in the following week. The regularization methods (3) can help further improve such data-driven forecasts.

In agreement with [14], our results emphasize the importance of benchmarking complex forecasting models against simple forecasting baselines to further improve forecasting accuracy. Our study also points towards recent findings on algorithm rejection and aversion [15] that found that “people have diminishing sensitivity to forecasting error” and that “people are less likely to use the best possible algorithm in decision domains that are more unpredictable”. Finally, in highly uncertain and noisy forecasting regimes, simple methods tend to outperform more complex methods because of a more favorable bias-variance tradeoff [16].