Probability Integral Transform: A Guide to Density-Forecast Calibration

A point forecast might say tomorrow’s return is expected to be 0.2%. A density forecast goes further: it assigns probabilities to the full range of returns, including ordinary moves and tail events. Its evaluation therefore asks whether the probability distribution issued before the outcome was observed is consistent with what later happened. The Mincer–Zarnowitz guide addresses linear calibration of numeric point forecasts, a different question from checking a full predictive distribution.

The PIT is a diagnostic for that distribution. At each forecast origin, it asks how much probability the forecast placed at or below the realized value. The value is a probability rank, not a return, a trading signal, or a measure of profit. In finance, a return-density model may feed risk estimates or decisions, but the PIT evaluates its distribution forecasts. The Diebold–Mariano guide compares average forecast losses under a chosen score; it answers a different question from checking density calibration.

Let (Y_{t+1}) be the outcome after forecast origin (t), and let (F_{t+1}(y\mid\mathcal I_t)) be the forecast cumulative distribution function (CDF), based only on the information set \(\mathcal I_t\) available at that origin. For a continuous forecast distribution, calculate

Uₜ₊₁ = Fₜ₊₁(Yₜ₊₁ | 𝓘ₜ)

The CDF is the probability assigned to values no greater than (y). If the forecast CDF is the true conditional distribution of the outcome, then the probability integral transform theorem gives

Pr(Uₜ₊₁ ≤ u | 𝓘ₜ) = u, for 0 ≤ u ≤ 1

Thus each ideal continuous one-step forecast produces a conditionally uniform PIT value. In a sequential setting where the information set includes the past, the ideal PIT sequence is also independent across origins under the maintained conditions. The transform itself is associated with Rosenblatt’s work; Diebold, Gunther, and Tay made PIT sequences operational for density-forecast evaluation (Rosenblatt, 1952; Diebold, Gunther, and Tay, 1998).

For a hand check, suppose the forecast at one origin is (N(0,1)) and the observed value is (y=1). Its PIT is (\Phi(1)\approx0.8413): that forecast assigned about 84.13% probability to outcomes at or below 1. This single value illustrates the calculation. It does not tell us whether the model is calibrated; that requires a sequence of forecasts and outcomes.

For a discrete forecast, the CDF jumps at possible outcomes. The ordinary value (F(Y)) therefore lands on a small set of CDF levels and is not generally Uniform(0,1), even when the forecast is correct. A randomized PIT fills each jump interval:

Uₜ₊₁ = Fₜ₊₁(Yₜ₊₁⁻ | 𝓘ₜ) + Vₜ₊₁ [Fₜ₊₁(Yₜ₊₁ | 𝓘ₜ) − Fₜ₊₁(Yₜ₊₁⁻ | 𝓘ₜ)]

Here (F(y^-)) is the left limit, or the probability of a value strictly below (y), and (V) is an independent Uniform(0,1) draw used only for evaluation. For a continuous distribution the jump is zero, so the expression reduces to (F(Y)). Brockwell proves the randomized transform for arbitrary distributions, including discrete and mixed cases (Brockwell, 2007). For ordered count outcomes, Czado, Gneiting, and Held also discuss a nonrandomized PIT version and marginal calibration diagrams; such discrete-specific diagnostics should not be judged by pretending ordinary \(F(Y)\) values are continuous-uniform (2009).

Consider a hypothetical Bernoulli forecast with (P(Y=1)=0.20), so (P(Y=0)=0.80). The ordinary PIT is 0.80 when (Y=0), an event with probability 0.80, and 1.00 when (Y=1), an event with probability 0.20. Those two values are not uniform. With randomization, (Y=0) maps to (0.80V), filling ([0,0.80]), while (Y=1) maps to (0.80+0.20V), filling ([0.80,1]). The first interval occurs with probability 0.80 and the second with probability 0.20; together their probability density is one across ([0,1]). The random draw is an evaluation device, not part of the original forecast.

A uniform marginal distribution of PIT values is necessary for ideal continuous density forecasts, but it is not sufficient. Pooling can hide errors that offset one another: one regime may receive forecasts that are too high and another forecasts that are too low, while the overall histogram looks flat. Gneiting, Balabdaoui, and Raftery discuss examples where an essentially uniform PIT histogram coexists with biased individual forecasts (2007).

Order also matters. A series can have a uniform histogram and still show serial dependence, such as alternating clusters or long runs of similar PIT values. Correct one-step conditional forecasts imply more than pooled uniformity when the forecast information set properly includes prior information: they imply that the PIT is uniform relative to that information. For overlapping multi-step horizons or other dependent forecast designs, however, the reference distribution and independence condition must reflect the overlap; an iid test is not automatically appropriate.

A useful diagnostic therefore checks both the PIT distribution and its relationship with time, lagged PITs, forecast-origin variables, and prespecified regimes. Grouped plots or conditional tests can reveal patterns that pooling conceals, but they introduce their own choices and finite-sample limits. No finite collection of chosen checks proves calibration for every possible conditioning variable.

A PIT histogram summarizes the marginal shape after values are placed into bins. A U-shaped pattern is often consistent with forecasts that are too concentrated or underdispersed; a central hump is often consistent with forecasts that are too diffuse or overdispersed. A left-right imbalance can be consistent with systematic location bias. These are diagnostic heuristics, not unique diagnoses: dependence, mixtures of regimes, discrete outcomes, binning, and small samples can produce misleading shapes.

The plot also discards the ordering of observations. It cannot show when a calibration problem began, whether large PIT values cluster, or whether a subgroup is miscalibrated. A flat-looking histogram may result from wide bins and limited data, while narrower bins can look irregular from sampling noise alone. Report the number of forecasts, bin boundaries, and uncertainty where feasible. Examine serial dependence and conditional slices chosen before looking for favorable results. For formal distribution checks, Diebold–Gunther–Tay’s framework and Berkowitz’s transformed-PIT likelihood-ratio approach are examples of tests with assumptions that must be matched to the forecast design. Berkowitz transforms an interior PIT with \(Z_t=\Phi^{-1}(U_t)\) and tests the joint null of independent standard-normal values against a specified dynamic alternative, such as an AR(1). Its conclusion depends on that test design; neither a plot nor one test is a universal certificate (Berkowitz, 2001).

<!-- learn:illustration -->

The uniformity result assumes the forecast CDF is the correct conditional law. In practice, a model estimates parameters, chooses a distribution family, and may tune thresholds or features. The estimated forecast (F_{t+1}(\cdot;\hat\theta_t)) is therefore part of a forecasting procedure, not a known true CDF. Reusing outcomes from the evaluation period to fit parameters and then describing the resulting PITs as untouched out-of-sample evidence leaks information into the check.

For a rolling evaluation, make each forecast with information available at its origin. Record the estimation window, re-estimation frequency, data vintage, model version, and any model-selection steps. Changing those choices changes the forecasting procedure being evaluated. A rolling or expanding window can also make forecasts statistically dependent because adjacent estimates share observations; overlapping multi-step outcomes add another source of dependence.

The impact on a formal test’s size and power depends on the estimator, sample, and testing procedure. A PIT histogram does not account for parameter uncertainty, and textbook iid reference values should not be assumed for every estimated, selected, or overlapping forecast design. When inference is important, use a test whose assumptions fit the design or simulate/bootstrap its null by repeating the full estimation and forecast pipeline. Keep a final evaluation sample untouched by model selection whenever the research question calls for genuine out-of-sample assessment.

Calibration concerns the joint behavior of forecasts and outcomes: do events assigned a given probability occur at the corresponding rate, and do forecast distributions align with what is observed? Sharpness describes the concentration of the predictive distributions themselves, without looking at outcomes. Gneiting, Balabdaoui, and Raftery argue that sharpness is desirable subject to calibration, not a substitute for it.

A broad forecast centered on the right distribution may be calibrated but not very informative. A narrow forecast may look precise, yet be poorly calibrated if outcomes fall outside its mass too often. A PIT assessment focuses on distributional consistency; sharpness summaries describe spread, interval width, or concentration. A forecast can improve one and worsen the other, so reporting just the histogram or just average interval width leaves out part of the picture.

Calibration language also depends on the question. A pooled PIT distribution can look uniform across all origins while forecasts miss conditional patterns by volatility state or forecast horizon. If those conditions matter for the intended use, define them in advance and inspect them. This is related to, but not the same as, a regression for point forecasts or the Giacomini–White conditional predictive-ability test, which compares forecast losses conditional on a chosen information set.

The PIT is primarily a diagnostic; it does not produce a single ranking of competing density forecasts. Proper scoring rules such as the logarithmic score and continuous ranked probability score (CRPS) assign a scalar loss to each forecast and outcome. Under their definitions, expected loss is minimized by reporting the true predictive distribution. They can compare forecasts on a common out-of-sample sample, but one realized average does not prove that a model is correct. The Model Confidence Set guide covers set-based comparison of forecasting methods. To compare density forecasts, use a proper score defined for the full distribution and apply it to common outcomes.

A score comparison should state the forecast target, horizon, common dates, loss convention, and benchmark. The log score is especially sensitive to assigning very small density to an observation; CRPS measures differences between cumulative distributions and has a different sensitivity profile. Report score differences with uncertainty that respects serial dependence, estimation, and any model search. Use PIT plots alongside scores: diagnostics can show a distributional defect, while scores summarize comparative performance under a stated penalty.

Neither calibration nor a better proper score by itself establishes trading profitability. To make a trading claim, separately define a decision rule, information timing, position sizing, transaction and funding costs, and an out-of-sample return evaluation. A forecast-evaluation statistic is not a backtest return.

Start by fixing the forecast target, horizon, origin time, outcome vintage, and whether the distribution is continuous, discrete, or mixed. Save every predictive CDF or enough information to reproduce it, alongside the realized value and the information set. For each origin, calculate (F_t(Y_t)) when the forecast is continuous; for jumps, use a documented randomized PIT or a discrete-forecast diagnostic designed for that setting.

Next, inspect the marginal distribution and time order. State histogram bins, sample size, treatment of ties, and random seed or repeated-randomization procedure for discrete forecasts. Add independence or conditional checks that answer a prespecified research question, and use inference calibrated for the actual rolling, overlapping, or estimated forecast scheme. Compare models separately with proper scores on identical held-out outcomes. The result is evidence about a defined set of forecasts and conditions, not proof that a model will remain calibrated or that a trading strategy will be profitable.