Diebold–Mariano Test Explained: Comparing Forecast Accuracy

The Diebold–Mariano test compares two forecasts of the same target over the same evaluation dates. At each origin, both procedures must face the same realized outcome, forecast horizon, and information cutoff. The test then asks whether the average loss of one forecasting procedure differs systematically from the other, allowing the loss differences themselves to vary over time.

Diebold and Mariano formulate the question for a general loss function, not just mean squared error. Their original paper develops tests of the null hypothesis that competing forecasts have equal predictive accuracy (Diebold and Mariano, 1995). In this setting, “accuracy” means lower expected loss under the loss function chosen for the comparison. It does not mean a forecast is true, causally explanatory, well calibrated in every part of its distribution, or useful for every decision.

Suppose model A predicts a future return and model B predicts that same return at the same time. The DM test is not a test of whether a coefficient in either model equals zero. It is also not a test that the two models have the same fitted values on the estimation sample. It compares how their forecast errors translate into the chosen loss on the evaluation sample.

The design has to be specified before interpreting the statistic. The target, forecast origins, horizon, loss function, treatment of missing outcomes, model-refitting schedule, and one- or two-sided alternative determine the question being tested. If these choices change between models, their loss difference no longer represents a like-for-like comparison.

Let the realized target at forecast origin t be $y_t$, and let model A and model B issue forecasts $\hat y_{A,t}$ and $\hat y_{B,t}$. Define their forecast errors as $e_{A,t}=y_t-\hat y_{A,t}$ and $e_{B,t}=y_t-\hat y_{B,t}$. For a loss function $L$, define the date-level loss differential as:

$$ d_t=L(e_{A,t})-L(e_{B,t}) $$

This sign convention is useful because a negative average $d_t$ means model A has lower observed loss, while a positive average means model B has lower observed loss. The population null is $E[d_t]=0$. A two-sided alternative asks whether expected losses differ in either direction; a one-sided alternative asks in a direction selected in advance. Do not choose the direction after seeing which model won.

With squared-error loss, $L(e)=e^2$, large misses count disproportionately. With absolute-error loss, $L(e)=|e|$, the ranking is less dominated by a single large miss. Quantile loss can match an asymmetric forecast objective, and other losses can target other aspects of predictive performance. Because a different loss can reverse the model ranking, “best forecast” has no unique meaning until the loss is named. Tashman’s review of out-of-sample forecasting tests likewise stresses that evaluation choices need to match the forecasting problem (Tashman, 200000065-0)).

Use the same forecast origins and outcomes for both models. If one model is missing a difficult date and the other is not, silently dropping that date only for the first model changes the comparison sample. If the intended deployment would refit a model every month, generate the evaluation forecasts with that refitting rule; a fixed-parameter forecast exercise answers a different question. A sequential evaluation such as walk-forward validation defines how later forecasts can be produced without using future data.

Assume four hypothetical forecast origins, and measure both the target and forecast errors in percentage points. The errors for model A are $[1,2,0,1]$; those for model B are $[2,1,1,1]$. Each pair refers to the same realized return and the same forecast interval. These values are invented only to show the arithmetic.

Under squared-error loss, model A has losses $[1,4,0,1]$, so its mean squared error is $(1+4+0+1)/4=1.50$ square percentage points. Model B has losses $[4,1,1,1]$, for a mean squared error of $(4+1+1+1)/4=1.75$. On these four outcomes, A's MSE is lower by 0.25 square percentage points.

The date-level differentials $d_t=L(e_{A,t})-L(e_{B,t})$ are $[-3,3,-1,0]$. Their mean is $(-3+3-1+0)/4=-0.25$, the same MSE difference computed as mean loss A minus mean loss B. The negative sign favors A under this convention. It says nothing yet about whether the difference is larger than the sampling noise. Four forecast pairs are not enough to treat the sign as persuasive statistical evidence.

The loss definition can also change what “better” means. For a separate hypothetical comparison, model C has absolute errors $[0,0,0,3]$ and model D has absolute errors $[1,1,1,1]$. Under absolute loss, C averages 0.75 while D averages 1, so C is preferred. Under squared loss, C averages 2.25 while D averages 1, so D is preferred. A test cannot resolve this disagreement without a decision about which forecast mistakes matter more.

<!-- Illustration placement: after this section; text-free conceptual scene showing two forecast paths against one realized path, with visibly different miss distances and no labels or numerical claims. -->

The sample mean loss differential is $\bar d=T^{-1}\sum_{t=1}^{T}d_t$, where $T$ is the number of matched forecast outcomes. Its standard error depends on the long-run variance of $d_t$, not just its variance on a single date. A general form of the test statistic is:

$$ DM=\frac{\bar d}{\sqrt{\widehat{\Omega}/T}} $$

Here, $\widehat{\Omega}$ estimates the long-run variance of the loss-differential series. If dates were independent, this would reduce to the variance of $d_t$ under the chosen estimator. Forecast losses often cluster, however: a period of high volatility can make both models' errors larger, and an $h$-step target can cause adjacent error windows to share realized returns. Ignoring positive dependence can make the estimated standard error too small and the apparent evidence too strong.

One common HAC construction estimates autocovariances $\hat\gamma_k$ of the centered loss differential and combines them as $\widehat{\Omega}=\hat\gamma_0+2\sum_{k=1}^{q}w_k\hat\gamma_k$. With Bartlett weights, $w_k=1-k/(q+1)$ through bandwidth $q$. Newey and West developed a positive-semidefinite heteroskedasticity- and autocorrelation-consistent covariance estimator (Newey and West). The kernel and bandwidth are implementation choices; disclose them and use values that reflect the forecast design rather than tuning them to obtain a preferred p-value. See the HAC standard-error guide for the broader covariance-estimation problem.

Under the test's regularity conditions, the standard DM statistic is compared with an approximately standard normal reference distribution. A large absolute value indicates that the observed mean loss gap is large relative to its estimated uncertainty. The statistic's sign identifies which model had the lower loss under the defined ordering; the p-value does not measure the size or economic value of the loss difference. As with other hypothesis tests, a p-value is not the probability that the null is true or that a forecast will work in the future. The p-value guide explains that distinction.

When forecasts are made every period for a target several periods ahead, the evaluation windows overlap. For example, a monthly forecast of the cumulative next-three-month return shares two of its three monthly returns with the forecast issued one month later. Even if the one-month returns were independent, this overlap can induce serial correlation in forecast errors and in their loss differences.

For a basic $h$-step setup, dependence through at least lag $h-1$ is a natural part of the variance calculation. That is not a universal bandwidth rule: rolling parameter estimates, persistent targets, volatility clustering, and the chosen loss can create additional dependence. Record the forecast horizon, origin spacing, and the reason for the HAC truncation or other variance estimator. The number of forecast rows is not automatically the number of independent pieces of evidence.

The original asymptotic test can be poorly sized in moderate samples. Harvey, Leybourne, and Newbold proposed a finite-sample modification for comparing mean squared forecast errors (HLN, 199700719-4)). A common form for horizon $h$ and $T$ forecasts rescales the DM statistic by:

$$ DM_{HLN}=DM\sqrt{\frac{T+1-2h+h(h-1)/T}{T}} $$

The modified statistic is commonly referred to a $t$ distribution with $T-1$ degrees of freedom. For $T=60$ and $h=5$, the multiplier is about $0.925$. This example only illustrates the correction's arithmetic; it does not say that 60 observations are sufficient for a particular model or that the assumptions are satisfied. The correction improves a specified small-sample comparison; it does not repair leakage, an invalid target, unmodeled dependence, repeated model selection, or a misspecified loss.

Model A may be a restricted version of model B. For example, B may add predictors to A. Under a null in which the extra predictors have no population forecasting value, their estimated coefficients can still add noise to B's forecasts. The larger model may therefore have a higher observed MSE even when its extra predictors do not improve the underlying forecast process. The usual equal-accuracy reasoning for non-nested competitors does not automatically apply unchanged.

Clark and West develop approximately normal tests for equal predictive accuracy in nested models and adjust the squared-error comparison for the estimation noise created by the larger model (Clark and West, 2007). Their adjustment is not a universal replacement for every DM test: it is designed for a particular nested-model question, loss, and asymptotic setup. When one model nests another, identify that fact and choose a test whose null distribution matches the design. Do not take a standard DM p-value from software and assume it covers every model relationship.

Related distinctions matter too. A forecast that improves the mean squared error may not improve interval coverage, probability calibration, directional accuracy, or a downstream decision. A forecast comparison can also be out of sample while still using an invalid holdout that was repeatedly inspected during model development. State the models' relationship, estimation scheme, forecast horizon, and what data were used to make each forecast.

A DM result evaluates a predictive loss. A trading decision uses a payoff and risk process. Lower MSE for next-period returns does not guarantee that a signal takes the right side of a trade often enough, sizes positions appropriately, or survives turnover, spread, market impact, fees, borrow, funding, and execution delay. Conversely, a forecast with slightly worse average squared error might still improve a decision if it is more accurate where the strategy has high exposure. The relevant loss for that claim may need to reflect the actual decision, not a convenient generic forecast metric.

Even a statistically significant difference can be economically tiny. Report the size of the mean loss difference in interpretable units alongside uncertainty, not only a p-value or a winning label. If the downstream claim is about portfolio results, replay the complete frozen decision rule on suitable later data, with realistic trading assumptions, and report net outcomes separately. DM does not compute those outcomes or account for costs unless the loss was explicitly constructed to do so.

The test also does not correct for searching many models, targets, horizons, loss functions, date ranges, or trading rules and then reporting only the most favorable comparison. Repeated pairwise tests create their own selection problem. Keep the search record and apply a multiple-testing method suited to the family of claims; the guide to multiple testing and false-discovery rate discusses why ordinary cutoffs can mislead after broad search. A DM test is an evaluation tool, not a correction for data snooping.

A clear report names the target, units, information cutoff, forecast origins, horizon, refitting schedule, and common evaluation sample. It states the loss function and sign of $d_t$, gives each model's average loss and their mean differential, identifies the one- or two-sided hypothesis chosen in advance, and reports the variance estimator, kernel, bandwidth, test statistic, reference distribution, p-value, and a confidence interval or uncertainty estimate where appropriate.

Also disclose whether the models are nested, how missing outcomes and extreme values were handled, how many alternative specifications were examined, and whether the evaluation period influenced model choices. Show the magnitude of the difference, not merely whether it passed a threshold. A time series of $d_t$ or a plot of cumulative loss differences can reveal regime changes that one overall average conceals, although the display does not replace dependence-aware inference.

In quantitative trading, add the map from forecast to action: signal threshold, position sizing, rebalance timing, risk controls, execution price, and cost assumptions. The DM test only compares the forecast-loss series you give it. It cannot certify the data pipeline, make two different evaluation samples comparable, prove causality, or promise that historical rankings will persist. Use it as one transparent part of model evaluation, together with a sound time-ordered forecast design and an explicit decision-level assessment.