White’s Reality Check vs. Hansen’s SPA Test for Trading Rules

If you test one strategy that was specified before looking at the sample, an ordinary single-comparison procedure may fit the question. But if you try many rules, inspect their results, and then test only the winner as if it had been chosen in advance, the reported evidence ignores the search. Even when every rule has no true advantage, the largest sample result will usually be positive just because it is the largest of several noisy estimates.

Reality Check (RC) and SPA address a family-wide question: after accounting for a defined set of alternatives, is there evidence that at least one has superior expected performance to a benchmark? The family may be technical rules, model forecasts, or other candidate strategies. White introduced the data-snooping framework (2000 paper); Sullivan, Timmermann, and White applied it across a large universe of technical trading rules (1999 application). The point is to include the alternatives that generated the selected result, not just the survivor. The false-discovery-rate guide covers a related but different multiple-testing question: FDR procedures adjust a collection of individual rejections, while RC and SPA test one family-wide maximum against a benchmark.

Let k index each candidate strategy in the tested family K, and let t index the same evaluation dates. For a return-based example, define the per-period difference as:

d(k,t) = net return of candidate k at t − net return of benchmark at t

A positive value favors the candidate. If the question is forecast accuracy, define the difference with the opposite loss order—benchmark loss minus candidate loss—so positive still means the candidate is better. Do not mix signs across candidates or dates.

For each candidate, let μ(k) = E[d(k,t)]. The family-wide hypotheses are:

H0: max over k in K of μ(k) ≤ 0 H1: at least one candidate k in K has μ(k) > 0

This is one joint test of the family, not a collection of independent tests that can be read one result at a time. The return definition, benchmark, dates, candidate set, and performance criterion must be explicit. If “better” means risk-adjusted performance, a mean-return difference alone is not that criterion.

Write d̄(k) for the average difference over n evaluation dates. In the simple mean-difference setup, the RC statistic is:

T_RC = max over k in K of sqrt(n) × d̄(k)

It takes the strongest sample result across the candidates and compares that maximum with a bootstrap distribution under the null. White’s procedure uses the least-favorable null configuration for this composite hypothesis: the null distribution treats every candidate’s expected difference as zero, even though candidates with negative expected differences also satisfy H0. That choice protects the test but can raise the critical value when the candidate family contains many weak alternatives.

The maximum is important. The question is not “Would the observed winner pass a single-rule test?” It is “Would a maximum at least this large be unusual after the same search over the stated family?” Testing only the winner against a single-rule reference distribution answers the first question and misses the selection step.

At each date, the candidate differences share market information and are often correlated with one another. The sequence may also have serial dependence. A bootstrap should resample the time-indexed vector of differences jointly so that each replicate preserves contemporaneous relationships across candidates and plausible time dependence within the series. Resampling each candidate independently would break those relationships and change the distribution of the maximum.

White’s test and Hansen’s implementation can use block methods such as the stationary bootstrap. The stationary bootstrap builds a pseudo-series by joining blocks whose lengths are random; Politis and Romano’s construction uses geometrically distributed block lengths in its standard form. The block design still needs to fit the sample’s dependence structure. The asymptotic results rely on regularity conditions such as a stable, weakly dependent process; a structural break or strong nonstationarity is not cured by resampling. A block that is too short can erase important persistence; a very long block leaves few effectively distinct chunks. State the resampling method and block-length setting, and check whether conclusions move when reasonable settings change.

The bootstrap must also represent the null being tested. In practice, implementations center or otherwise transform the candidate difference series to construct a null distribution, then recompute the maximum statistic in every replicate. The technical details differ between RC and SPA. A generic confidence interval bootstrap for one already-selected strategy does not automatically perform this family adjustment. The block-bootstrap guide explains why dependence-preserving resampling differs from shuffling individual returns.

Hansen’s SPA test keeps the same family-wide null and benchmark-relative differences, but changes two parts of the procedure. First, it studentizes each candidate’s mean difference using an estimate of the long-run standard deviation of its differential series:

T_SPA = max(0, max over k in K of sqrt(n) × d̄(k) / ω̂(k))

This expresses candidates in standard-error units rather than comparing raw means with different variability. The long-run scale reflects serial covariance across dates; it is not necessarily the ordinary standard deviation of the observed returns. The variance estimator and any lag or bandwidth choices should be reported because they affect the standardization. Second, SPA uses a sample-dependent null distribution. It retains candidates that appear compatible with the null while reducing the influence of alternatives that the data indicate are clearly poor. In the consistent version, sufficiently negative sample means are carried into the estimated null, while candidates near its boundary are centered at zero; the procedure does not simply delete poor candidates. This is designed to avoid the RC’s least-favorable treatment of every candidate as exactly tied with the benchmark.

Hansen’s paper reports that these modifications can improve power and reduce sensitivity to poor or irrelevant alternatives. That is not a theorem that SPA wins in every setting: Hansen notes that the tests do not uniformly dominate one another. Some implementations also report lower, consistent, and upper SPA p-value variants; name the version and implementation used rather than treating every “SPA p-value” as identical. Hansen’s paper gives the studentized statistic and sample-dependent bootstrap details.

Suppose a researcher tries 12 lookback windows, 4 entry filters, and 5 exit settings, then selects the best-performing combination. That is a hypothetical family of 240 candidates if every combination was evaluated. An RC or SPA rejection can support the statement that the stated family contains evidence of superior performance relative to its benchmark under the chosen criterion and assumptions.

It does not establish that the selected combination is individually significant, that it is the best live rule, or that every alternative was included. A global maximum test is not a post-hoc ranking procedure. If the goal is to make adjusted claims about specific rules, use a method designed for stepwise or candidate-level inference, then validate the selected rule on data that did not help choose it. A non-rejection also does not prove that all rules are useless; it says the sample did not provide enough family-wide evidence under this test.

The test can only adjust for the family supplied to it. Include tried parameter settings, filters, markets, holding periods, and variants that influenced selection, including unsuccessful ones. If several benchmarks, return definitions, or sample windows were also tried and the preferred version was chosen after inspecting results, record that search too. An undocumented research path remains outside the formal correction.

Use returns aligned on the same dates and apply realistic transaction costs, financing, borrow, and roll assumptions before defining the performance difference. Select the criterion before looking at the winner. Mean net return, a utility score, and a risk-adjusted measure are different questions; a test built for one does not silently answer the others. In nested forecasting comparisons or other special designs, assumptions about parameter estimation and the null distribution can differ, so a general RC/SPA recipe should not be applied automatically.

RC and SPA can be reported as complementary checks when the research question and candidate family support both. Decide which test is primary before inspecting the results, then explain any secondary statistic as a sensitivity analysis. If their conclusions differ, investigate whether studentization, poor alternatives, dependence estimates, or the SPA recentering variant explains the difference. Do not choose the test after seeing which one produces the smaller p-value; that choice is another unrecorded search.

The same principle applies to the performance measure and bootstrap design. A sensitivity analysis can show how conclusions respond to defensible alternatives, but it does not turn a collection of favorable specifications into independent confirmation. State the full set of reported results and preserve the decision path so readers can distinguish a planned robustness check from a selected headline result.

A useful report names the benchmark, candidate family, performance criterion, date range, observation frequency, costs, bootstrap method, block-length choice, number of resamples, statistic, and exact p-value variant. Preserve the tested family so someone else can see whether the alternatives that led to selection are represented. If an untouched chronological holdout is used after the test, do not tune the rule on that holdout and continue calling it untouched.

An RC or SPA p-value is evidence about a family-wide null conditional on the data, search universe, statistic, and assumptions. It is not the probability that a particular strategy will make money, a forecast of future returns, or a guarantee that backtest profits survive costs and regime change. The test addresses one part of model selection; it does not fix look-ahead bias, survivorship, data errors, market impact, or an incomplete search record. Purged time-series cross-validation addresses leakage in a different validation question.