The Model Confidence Set (MCS): Compare Forecast Models Without Picking a Winner

Suppose you have several procedures that forecast next-day volatility. Their errors are not identical, but the sample may be too short or noisy to tell which procedure has the smallest expected loss. Selecting whichever one has the lowest average loss would force a winner even when the gap could be sampling variation. The Model Confidence Set offers a set-valued answer: which candidates remain statistically compatible with being best under a specified performance criterion?

The MCS starts with a candidate set \(\mathcal M_0\), a loss or other criterion, and a confidence level. It repeatedly tests whether the candidates still under consideration have equal predictive ability. If the test rejects, an elimination rule removes a candidate judged relatively poor; testing continues on the reduced set. The survivors form the MCS. Hansen, Lunde, and Nason introduce this procedure as a confidence set for the best model or models in the stated collection, much as a parameter confidence interval can retain more than one value when the data are imprecise (2011 paper).

“Best” is local to the comparison. A model is best only relative to the candidates included, the target, the forecast horizon, the information available at each origin, and the criterion used to score it. The MCS does not require one candidate to be the true data-generating process, and it can retain multiple candidates that have the same best expected performance. It is not a posterior probability that a retained model is correct.

Before calculating losses, define the set \(\mathcal M_0\). A candidate can be a fitted model together with its estimation and forecast-update rule, or a forecasting procedure that is not described by a statistical model. For example, “GARCH” is not fully specified if the volatility distribution, rolling window, parameter updates, and forecast horizon can all change. Each of those choices can produce a different candidate.

Every candidate needs to forecast the same target for the same origins and horizon using only information that would have been available then. If one model forecasts one-day variance and another forecasts five-day volatility, their raw losses are not a fair head-to-head comparison. Use a common evaluation sample, align missing observations explicitly, and record the data vintage, initial estimation window, recursive or rolling schedule, and treatment of revised inputs. When forecasts overlap in time, neighboring losses can share observations even though they were issued at different origins.

Choose the criterion before comparing outcomes. For a point forecast, squared error, absolute error, or a decision-specific loss can rank models differently. A variance forecast may use a volatility loss suited to a noisy proxy; a quantile forecast needs a quantile score rather than mean squared error. A good MCS under one loss does not imply a good MCS under another. If the candidate list was narrowed after inspecting the same evaluation results, the formal set is conditional on that unreported screening and no longer reflects the full search.

Let \(y_{t+h}\) be the realized target for a forecast made at origin \(t\), and let \(\hat y_{i,t+h|t}\) be candidate \(i\)'s forecast. For a preselected loss function \(L\), calculate one loss for every candidate and common origin:

\[ L_{i,t}=L(y_{t+h},\hat y_{i,t+h|t}) \]

Assume lower loss is preferred. For candidates \(i\) and \(j\), define the paired contrast

\[ d_{ij,t}=L_{i,t}-L_{j,t}, \qquad \bar d_{ij}=\frac{1}{P}\sum_{t=1}^{P}d_{ij,t} \]

where \(P\) is the number of common forecast origins. Under this sign convention, a positive \(\bar d_{ij}\) means candidate \(i\) had higher average loss than \(j\) in the sample; a negative value favors \(i\). Pairing matters because both candidates faced the same realized outcomes. Common market shocks can raise both losses, while the contrast isolates their relative performance.

The equal-predictive-ability null for a current set \(\mathcal M\) can be written as

\[ H_{0,\mathcal M}:\quad \mu_{ij}=E[d_{ij,t}]=0 \quad \text{for every } i,j\in\mathcal M \]

This is a joint statement about the current set, not a collection of unrelated pairwise tests that can be inspected one at a time. The original MCS framework uses a joint equivalence test and an elimination rule to manage the search across models. It does not ask whether the forecasts are exactly equal on each date.

The MCS procedure alternates between a set-level test and an elimination step. Begin with \(\mathcal M=\mathcal M_0\). Test equal predictive ability at the chosen level \(\alpha\). If the test does not reject, stop and report the current set. If it rejects, use a prespecified elimination rule to remove one candidate and repeat the test on the smaller set. The surviving set is the \((1-\alpha)\) MCS, subject to the procedure's assumptions.

The paper develops two familiar statistics. The range statistic looks for the largest standardized pairwise loss contrast:

\[ T_{R,\mathcal M}=\max_{i,j\in\mathcal M}|t_{ij}|, \qquad t_{ij}=\frac{\bar d_{ij}}{\widehat{\mathrm{se}}(\bar d_{ij})} \]

Its matching elimination rule removes the candidate with the largest standardized disadvantage against another candidate. A second statistic compares each candidate with the average performance of the current set:

\[ T_{\max,\mathcal M}=\max_{i\in\mathcal M}t_{i\cdot}, \qquad d_{i\cdot,t}=\frac{1}{|\mathcal M|}\sum_{j\in\mathcal M}d_{ij,t} \]

Here \(t_{i\cdot}\) standardizes the mean of \(d_{i\cdot,t}\); its matching rule removes the candidate with the largest standardized excess loss relative to the set average. The statistic and elimination rule form a pair. They should be chosen and reported as a procedure, not mixed after seeing which one produces a smaller set. The MCS paper shows why the rejection test must be coherent with the rule that identifies a candidate for removal (Hansen, Lunde, and Nason).

The distribution of a maximum statistic depends on the joint variability of the candidates' loss contrasts. Treating each model or date as an independent observation can break both forms of dependence: successive losses can be serially related, and two candidates' losses on the same date are paired. The MCS bootstrap therefore has to preserve the relevant joint time-series structure while approximating the null distribution of the set-level statistic.

One approach is to resample aligned time blocks of the vector of candidate losses or contrasts. In a stationary bootstrap, blocks have random lengths and the resampled series can restart at a randomly selected date; Politis and Romano introduced this method for weakly dependent stationary observations (1994 paper). The MCS authors describe block bootstrap implementations and note the stationary bootstrap as an alternative in their implementation appendix. Independently resampling each model's loss series would destroy same-date pairing and can change the comparison.

The average block length, bootstrap variant, centering under the equal-ability null, number of replications, and forecast horizon all affect the resulting uncertainty estimate. Longer overlapping forecast horizons often create longer dependence in loss differences, but there is no universal block length that fits every target and sample. Explain the design and inspect whether the conclusion changes under reasonable dependence settings. Bootstrap output is an approximation under its assumptions, not a repair for look-ahead, a reused holdout, nonstationary score behavior, or a missing candidate family.

Suppose four procedures A, B, C, and D forecast the same volatility target at 120 common daily origins. Their average squared-error losses, in illustrative squared percentage-point units, are 1.20, 1.23, 1.45, and 1.90. Those averages rank A first and D last, but a ranking alone says nothing about sampling uncertainty.

For A versus B, the mean paired loss difference is \(1.20-1.23=-0.03\). If its dependence-aware standard error is \(0.04\), the corresponding pairwise statistic is \(-0.03/0.04=-0.75\). That single contrast does not show a statistically clear difference between A and B. The MCS does not stop there: its set-level statistic considers the joint contrasts among all four candidates.

For illustration, suppose a block-bootstrap MCS run on the full 120-origin loss series uses \(T_R\), \(\alpha=0.05\), and a coherent worst-candidate elimination rule. Assume it reports a set-level p-value of 0.018, removes D, and then reports 0.11 for \(\{A,B,C\}\). Since 0.11 exceeds 0.05, the procedure stops with \(\{A,B,C\}\) as the illustrative 95% MCS. C remains even though its average loss is higher than A's: this hypothetical run has not established that C is inferior under the joint test.

The p-values here are invented to demonstrate the steps; they cannot be reconstructed from the four average losses or the A-versus-B contrast. An actual result requires the complete origin-by-origin loss series, candidate forecasts, bootstrap design, and elimination sequence. In practice, preserve those inputs and report the full sequence so a reader can see which candidate left at each rejection. <!-- learn:illustration -->

At a 95% confidence level, the MCS is designed, under its regularity and testing conditions, to contain the best candidate in the specified set with at least the stated asymptotic coverage. It does not mean that every retained model has a 95% probability of being best. The confidence statement concerns the procedure over repeated samples; it is not a posterior distribution over model labels.

Failure to reject equal predictive ability is not proof that the candidates have exactly equal expected loss. The test may have limited power, especially with a short evaluation period, high volatility in the loss contrasts, or several near-tied candidates. A large surviving set can be an honest sign that the data do not separate the alternatives. It can also contain models that are all poor in absolute terms, because MCS compares candidates with one another rather than with a required external standard.

The set is also bounded by what entered \(\mathcal M_0\). If a genuinely better forecasting procedure was omitted, the MCS cannot discover it. If a model is removed before the analysis because an earlier look at the same evaluation period made it seem weak, the nominal coverage does not account for that unrecorded search. State the candidate-generation and screening history. Where the best expected loss is shared by several candidates, retaining more than one is consistent with the method rather than a failure to choose.

The Diebold–Mariano test focuses on a paired loss difference for two forecasting procedures. MCS repeatedly tests a set and reports which candidates survive its joint elimination procedure. Running many DM tests and keeping the nonsignificant pairs is not automatically equivalent to MCS; the joint bootstrap and coherent elimination rule matter.

The Clark–West test addresses a narrower squared-error comparison when one forecast model nests a restricted benchmark and estimation noise affects the raw difference. MCS does not require nested candidates and can use a user-specified loss, but it still needs a common forecast design.

White's Reality Check and Hansen's Superior Predictive Ability test ask whether at least one member of a searched family beats a designated benchmark. MCS is benchmark-free and describes a set of relatively superior candidates rather than testing one benchmark. These procedures all need a faithful account of the search and dependence, but they answer different null hypotheses. See the Reality Check and SPA guide and the original papers by White (2000) and Hansen (2005).

A reproducible MCS report names every candidate procedure, the common target and horizon, forecast-origin and information rules, estimation schedule, evaluation dates, missing-data treatment, loss formula, and which direction counts as better. Give the significance level, test statistic and elimination rule, bootstrap type, block-length choice, centering method, replication count, p-values at each step, and final membership. Report average losses and differences as context, but do not replace the joint inference with a table of rankings.

For volatility evaluation, state how the observed proxy was formed and whether it is noisy or revised. For multi-step predictions, disclose overlapping target windows and the dependence method. Show sensitivity to a plausible change in loss, sample period, and bootstrap settings when those choices materially affect membership. A single tiny p-value without the eliminated candidates and procedure details is not enough to reproduce the set.

An MCS ranks predictive procedures for one criterion. It does not establish causal structure, identify the data-generating model, or prove that the remaining forecasts create a profitable trade. Turning a forecast into a position adds thresholds, sizing, turnover, execution timing, spread, fees, slippage, financing, borrow, and risk constraints. Evaluate that frozen decision process on suitable later data and report net trading outcomes separately. For volatility scores under imperfect proxies, see the guide to QLIKE versus squared-error loss.