White’s Reality Check for Data Snooping in Strategy Backtests

A researcher may try dozens of moving-average windows, breakout rules, holding periods, markets, and filters, then report the strategy with the highest backtest score. That winner is not an ordinary single candidate: the search gave it many chances to look unusually good. Testing the winner alone as if it had been named before seeing the data ignores that selection step.

White’s Reality Check turns the selection into the test. It asks whether any candidate in the examined family has superior expected performance to a defined benchmark. White formulates the null as the intersection of one-sided comparisons: every candidate’s expected benchmark-relative performance is at most zero. The alternative is that at least one candidate has a positive expected advantage. The benchmark could be buy-and-hold, a simple forecasting rule, or another comparison rule, but its definition must match the research question and be set before reading the test result. See White’s original paper for the formal setup.

This is a family-level question, not a guarantee about the strategy that happens to win. Rejecting the null says the recorded family contains evidence of superiority under the chosen statistic and assumptions. It does not establish that every candidate is useful, that the winner will remain best, or that its advantage will survive a new market regime.

Let d[k,t] be candidate k’s performance minus the benchmark’s performance over the same period t, with a higher value always meaning better performance. For a return comparison, one possible definition is the candidate’s net return minus the benchmark’s net return. If the decision uses forecast loss instead, orient the difference as benchmark loss minus candidate loss. The sign convention must make a positive value favorable to the candidate.

Each row should refer to the same time interval across all candidates. Use one currency, return convention, financing treatment, and cost policy. If the claim concerns executable strategy returns, subtract the relevant commissions, spread, slippage, funding, and borrow costs before forming d[k,t]. Putting costs into a footnote while testing gross returns answers a different question.

The sample average for candidate k is d̄[k] = (1/T) Σ_t d[k,t]. The joint null is H0: max_k E[d[k,t]] ≤ 0; the alternative is H1: max_k E[d[k,t]] > 0. The benchmark is common across the candidate family, and every candidate is evaluated with the same criterion. If candidates have different missing dates or observation frequencies, define a defensible common comparison sample before calculating the differences rather than silently giving one candidate a more favorable window.

The family is the set of candidate rules whose results could influence the reported winner. It can include tested lookbacks, entry thresholds, signal combinations, asset universes, holding periods, portfolio constraints, and execution assumptions. It also includes manual variants that were tried and then discarded after their results were seen. White’s 2000 article uses “specification search” broadly: the choice process, not just the final table, creates the selection problem.

There are two opposite errors. Leaving out experiments that helped choose the reported strategy makes the adjusted test too optimistic because the bootstrap sees a smaller search than the researcher actually ran. Adding many arbitrary, unrelated rules after the fact can make the test needlessly conservative and reduce its ability to detect a useful candidate. Define the family from the actual selection process, and retain a research log that makes the boundary auditable.

The test cannot infer experiments that were never recorded. A notebook containing only the winning parameter set is not enough to reconstruct the search. Keep the candidate registry, data version, evaluation dates, objective, cost assumptions, and selection decisions together. If the family changed after examining results, state what changed and why; do not present a post hoc family as if it had been fixed in advance.

For each candidate, calculate its mean relative performance, then take the largest one. A common unstudentized statistic is Vobs = √T × max_k d̄[k]. The maximum is essential: it represents the same “pick the best” operation that produced the apparent winner. Testing only the winning column would remove that operation from the reference distribution.

The bootstrap approximates how large a maximum could arise from sampling variation if the joint no-superiority null were true. For bootstrap draw b, resample the time-indexed vector of all candidates’ differentials and compute a centered statistic such as V*b = √T × max_k(d̄*[k,b] − d̄[k]). Centering puts the candidate means at the least-favorable boundary of the null, where expected advantages are zero, while the maximum preserves the selection over candidates. White’s paper develops the bootstrap distribution of the maximum and compares the observed statistic with that distribution.

Repeat this process many times. The adjusted p-value is the share of bootstrap maxima at least as large as Vobs; with a finite number B of draws, one common Monte Carlo estimate is (1 + count{V*b ≥ Vobs})/(B + 1). Implementations may differ in studentization or details for estimated models, so document the statistic and null-centering convention. The key is that every bootstrap replicate repeats the candidate-family maximum, rather than evaluating only the observed winner.

Financial observations are ordered in time. An independent shuffle can erase serial dependence, volatility clustering, and runs of correlated gains or losses. It can also distort the relationship between two strategies that tend to respond to the same market days. Those features affect the tail of the maximum statistic, so resampling each candidate independently or drawing individual dates with replacement may produce the wrong reference distribution.

White describes dependent-data methods such as the moving-block bootstrap and analyzes the stationary bootstrap of Politis and Romano. In the stationary bootstrap, a sampled block usually continues to the next time observation; at a random restart it begins at another sampled date. The resulting block lengths are geometrically distributed, with a chosen average length. This preserves short-run sequences more faithfully than an i.i.d. resample, subject to the method’s assumptions and tuning choices. See Politis and Romano’s stationary-bootstrap paper.

For a strategy family, resample one shared sequence of time indices and apply it to the full row vector (d[1,t], …, d[K,t]). That retains both the time order inside sampled blocks and contemporaneous dependence among candidate strategies. Choose block length with the strategy horizon and dependence diagnostics in mind, and report sensitivity to reasonable alternatives. If model parameters are re-estimated as part of the research procedure, decide whether the inferential target includes that estimation step and reproduce it inside each resample when needed; resampling only fixed fitted returns may condition away part of the procedure.

Suppose a researcher compares three strategies with a benchmark over T = 252 trading days. After costs, their average daily differentials are +2.0, +1.0, and −0.5 basis points. The winner’s mean is therefore 2.0 basis points, and the observed statistic is √252 × 2.0 bp ≈ 31.75 bp·√trading day. The scaling is part of the test statistic; it is not a t-statistic because it is not divided by an estimated standard error.

Now suppose 9,999 stationary-bootstrap replicates use the same sampled dates for all three candidates. In this hypothetical output, 1,199 replicate maxima equal or exceed 31.75. The plus-one Monte Carlo estimate is (1 + 1,199)/(9,999 + 1) = 0.12. A separate, winner-only test might hypothetically report 0.03, but that would omit the search over three candidates. The Reality Check p-value uses the larger family comparison and, at a 5% threshold, this example would not reject the joint null.

These numbers are invented to show the calculation, not measured market performance or a result from White’s paper. A p-value of 0.12 does not mean there is a 12% chance the strategy will lose. It means that, under the specified bootstrap and null construction, a maximum at least this large is not rare enough to reject at the chosen level. The sample means also do not tell the reader whether the selected strategy’s return is economically meaningful after risk, capacity, and execution effects.

A small Reality Check p-value is evidence against the claim that no member of the included family beats the chosen benchmark in expected performance, under the test’s assumptions. Because the null is joint, it does not identify which candidate has a durable edge. The winner’s identity can change across samples, and the evidence for one strategy may be weak even when a family-level null is rejected.

A large p-value is a failure to reject, not proof that every strategy is equivalent to the benchmark. The result can reflect limited sample size, noisy performance, a very broad family, poor measurement, or low power. Likewise, the p-value is not the probability that the null is true. It is a tail probability computed under a reference distribution that depends on the candidate set, performance measure, benchmark, bootstrap design, and observed data.

White’s question is relative. A rule can beat a weak benchmark while losing money, or fail to beat a strong benchmark while still producing positive returns. Report absolute and benchmark-relative results together, along with uncertainty, turnover, drawdown, capacity, and realistic costs. Statistical evidence of relative predictive superiority and an investable economic case are separate decisions.

The original theory relies on regularity conditions for the performance-difference process and its limiting distribution; White’s setup includes a stationary, strongly mixing time series, and the dependent bootstrap needs a suitable block-length sequence. In finite samples, market regime changes, structural breaks, long memory, rare jumps, and unstable volatility can make a stationary resampling approximation questionable. A block bootstrap preserves some local dependence; it does not recreate an unknown future regime.

The test also inherits errors in the input data and strategy evaluation. Look-ahead leakage, survivorship, revised data, unrealistic fills, omitted costs, incorrect corporate-action handling, and a benchmark chosen after seeing the results can invalidate the performance differences. When strategy returns overlap across holding periods, the sequence may be dependent by construction; the resampling unit and block length must be able to represent that dependence. If the final metric is nonlinear, such as a Sharpe ratio, reproduce the metric calculation inside each replicate instead of assuming that a mean-return bootstrap automatically tests it.

Reality Check implementations can be conservative, especially when a large candidate set contains many clearly poor alternatives. A wide maximum distribution may make rejection difficult even when one good candidate exists. Hansen’s Superior Predictive Ability test is a related bootstrap approach with a different treatment of poor alternatives; it is not a universal replacement, and its assumptions still need review. Compare methods based on the research question and report sensitivity rather than selecting the one that produces the preferred answer.

White’s Reality Check addresses whether a recorded search family contains a candidate with benchmark-relative superiority after selection. It does not replace a chronological holdout or walk-forward evaluation of a frozen strategy. It does not solve label overlap or leakage by itself. It also differs from PBO, which summarizes how often an in-sample winner ranks below the candidate median in combinatorial splits; the original PBO paper formalizes that selection-risk diagnostic. False-discovery procedures target the expected false share among a set of rejections. The Deflated Sharpe Ratio instead adjusts a Sharpe-based significance assessment for selection and non-normal returns. See the guides to [multiple testing and false discovery in finance](/en/learn/multiple-testing-false-discovery-rate-finance-explained), [PBO and CSCV](/en/learn/probability-of-backtest-overfitting-cscv-explained), [walk-forward validation](/en/learn/walk-forward-validation-expanding-vs-rolling-windows-explained), and [purged cross-validation and embargo](/en/learn/purged-cross-validation-embargo-time-series-finance-explained).

In a practical review, freeze the benchmark and performance definition, reconstruct the actual candidate family, calculate paired period-by-period differences, choose and justify a dependence-aware resampling design, and report the maximum statistic and bootstrap tail probability. Then test the frozen selection process on later chronological data and assess costs and implementation separately. Sullivan, Timmermann, and White’s study of technical trading rules demonstrates why the universe of rules matters: the conclusion can change when the complete search, rather than only the reported winner, enters the inference. The Reality Check is a useful correction for a specific selection problem, not a certificate that a backtest will survive live trading.