Brier Score vs. Log Loss for Probability Forecasts

Imagine recording whether a model correctly called “up” or “down.” A hit rate treats a 51% forecast and a 99% forecast as the same correct call. It also treats a 49% and a 1% forecast as the same miss. That discards confidence, even though the confidence is the part that can matter when decisions have different payoffs or risks.

A binary probability forecast assigns a value (p_t) between zero and one to a clearly defined event at forecast origin (t). The event later resolves as (y_t=1) if it occurs and (y_t=0) if it does not. A scoring rule evaluates the forecast together with that outcome. The Brier score uses squared probability error; log loss uses the negative logarithm of the probability assigned to what actually happened.

These scores are useful for comparing probability forecasts, including event probabilities from a trading model. They do not by themselves show that a forecast is calibrated, that it beats a sensible reference, or that trading on it would make money. The Mincer–Zarnowitz guide covers a different linear calibration regression for numeric point forecasts.

Before scoring, state the event in a way that can be resolved consistently. “Will the asset rise?” is incomplete until the asset, price source, start and end times, currency, return convention, and handling of missing or corrected records are specified. For an economic event, record the release and vintage used to determine the outcome. Two forecasts should not be compared if they were scored against different definitions or date sets.

Store each forecast as it was available at its origin. A probability issued at time (t) must use information available by the stated cutoff, and it must be paired with the outcome for the same horizon. A revised data series, a later exchange close, or a classification rule chosen after observing the outcome can quietly change the target or introduce look-ahead information.

For rolling forecasts, preserve the model version, input data vintage, timestamp, probability, and resolution rule. Use a common set of origins when comparing models. If a probability is missing, document the exclusion and apply the same sample rule to every candidate. These records make a score reproducible rather than a summary of a changing spreadsheet.

For one binary event, the Brier loss on case (t) is

BSₜ = (pₜ − yₜ)²

The mean Brier score over (n) forecasts is

BS = (1/n) Σₜ (pₜ − yₜ)²

Lower is better, zero is perfect, and this single-event convention ranges from zero to one. For example, if the forecast is (p=0.70) and the event happens, the loss is ((0.70-1)^2=0.09). If the same forecast is followed by a non-event, it is ((0.70-0)^2=0.49). A confident miss costs more than a modest miss, but the penalty remains bounded.

Check the formula when comparing published numbers. Some conventions sum squared errors across every category of a multi-outcome forecast; a binary vector sum can be twice the single-event loss above. The ranking within one fixed event is unchanged by multiplying every score by two, but the numeric level is not comparable across conventions unless the scaling is stated.

For a binary event, log loss is

LLₜ = −[yₜ log(pₜ) + (1 − yₜ) log(1 − pₜ)]

If the event occurs, the loss is (-log(pₜ)); if it does not, it is (-log(1-pₜ)). A correct 70% forecast therefore receives (-\log(0.70)\approx0.357), while an incorrect 70% forecast receives (-\log(0.30)\approx1.204). The mean log loss averages these case-level penalties, so it has no fixed upper bound as a forecast assigns vanishing probability to the event that occurs.

At (p=0) or (p=1), a confidently wrong forecast has infinite log loss. If software clips probabilities to a small floor such as ε to avoid infinity, report the floor and apply it consistently before outcomes are examined. Clipping changes the numerical evaluation rule and should not be hidden as a data-cleaning detail.

The Brier score and log loss can rank the same forecasts differently because they express different penalty curves. Log loss puts especially high cost on assigning almost no probability to an event that occurs; the Brier score caps a binary case at one. Report which score was selected in advance and, when decisions depend on extreme probabilities, consider showing both rather than choosing the better-looking result afterward.

<!-- learn:illustration -->

A score is proper when the forecaster maximizes expected reward, or minimizes expected loss, by reporting the probability they actually believe. It is strictly proper when that truthful value is the unique optimum. The Brier and logarithmic scores are strictly proper for binary probability forecasts under their usual definitions (Gneiting and Raftery, 2007). This gives a principled reason to evaluate probabilities instead of converting them to hard labels first.

“Proper” is an expectation statement, not a promise about every realized sample. A forecaster who reports a truthful 70% probability can still be wrong on one case, and can have a worse finite-sample score than someone who guessed. Repeated, comparable forecasts are needed to learn about average performance. Incentives also matter: if a person faces a separate payoff rule, a score alone cannot guarantee their reported probability reflects their belief.

Neither score measures calibration alone. An aggregate can mix how close probabilities are to observed frequencies with how well they separate cases that resolve differently. A model can make a sharp set of forecasts and still be systematically miscalibrated; a model that always reports the base rate can be calibrated in aggregate while giving little case-specific information.

Murphy’s decomposition separates the mean Brier score into three terms (Murphy, 1973012%3C0595%3AANVPOT%3E2.0.CO%3B2)):

BS = reliability − resolution + uncertainty

For groups (k) of forecasts with similar probabilities, let (w_k) be each group’s share of the sample, ̅pₖ its mean forecast probability, ̅yₖ its observed event frequency, and ̅y the overall event frequency. Then the binned components are

reliability = Σₖ wₖ(̅pₖ − ̅yₖ)² resolution = Σₖ wₖ(̅yₖ − ̅y)² uncertainty = ̅y(1 − ̅y)

Lower reliability error is better: forecasts within a group should match the group’s observed frequency. Higher resolution is better: groups should have event frequencies that differ from the overall base rate. Uncertainty reflects how often the event occurs in the sample and cannot be credited to a forecasting model. The decomposition explains why two models can have the same Brier score but different strengths.

In practice, grouping continuous probabilities requires a binning choice. A reliability diagram based on ten equal-width bins can tell a different story from one based on quantile bins, especially in a small sample or near probability extremes. Show the bin counts and uncertainty, and treat the decomposition as a diagnostic rather than a way to erase sampling error. Calibration plots should not be mistaken for independent proof of future reliability.

A lower raw score than another model is a comparison; it is not yet a statement about improvement over a useful baseline. The Brier skill score compares a forecast system with a named reference forecast:

BSS = 1 − BS_model / BS_reference

A value of zero matches the reference, a positive value improves on it, and a negative value is worse under this definition. A value of one corresponds to zero model Brier loss when the reference loss is positive. Choose the reference to match the decision and information set. A fixed historical base rate, a training-window event frequency, or an existing forecast procedure may be sensible in different tasks.

Do not calculate the reference from future evaluation outcomes if it would not have been available when the forecast was made. For a time series, an expanding or rolling historical frequency can be a cleaner operational baseline than a full-sample event rate. If the reference score is zero, the ratio is undefined; state how the benchmark is formed and report the raw model and reference scores alongside BSS.

Brier skill scores depend on the reference and event frequency. A score against an unconditional base-rate forecast answers a different question from a score against a current production model. Do not compare BSS values across studies without checking their event definitions, reference forecasts, sample periods, and binary or multicategory conventions.

Generate probabilities chronologically and reserve an evaluation period that was not used to tune the model, choose features, or select a threshold. Score every model on the same resolved outcomes and origins. For a chosen loss, define the paired difference as

dₜ = Lₐ,ₜ − Lᵦ,ₜ

A positive mean under this sign convention means model B had lower average loss. The Diebold–Mariano framework can test an average loss difference when its forecast-comparison assumptions and variance estimate suit the design. If the question is whether the relative score changes with conditions known at forecast time, the Giacomini–White guide covers that conditional comparison. Repeated origins, overlapping event windows, and model search can make the differences dependent or selected, so a naive standard error or a single uncorrected p-value may overstate the evidence.

Choose the event, horizon, sample, primary score, benchmark, and comparison direction before viewing results. Report the mean Brier and log losses, their sample sizes, model and reference definitions, probability clipping rule, and any dependence or search adjustment. Showing a reliability plot and paired score differences alongside the aggregate helps explain what changed. Forecast combination methods can combine forecasts, but the final combination still needs its own untouched evaluation.

The scores are not in common units. A Brier difference of 0.01 is not directly the same magnitude as a log-loss difference of 0.01. A lower score also does not prove the underlying probability model is correct; it is evidence about the evaluated forecasts on the stated outcomes.

A probability can support a trading decision only through a rule that maps it to an action. The decision also depends on payoff size, losses when wrong, execution prices, spreads, commissions, slippage, funding, borrow, capital limits, and the time at which the forecast becomes tradable. A proper score evaluates a probability forecast; it does not include those costs or select a position size.

A model can improve average log loss without improving a particular trading rule, because the rule may trade only one probability range or respond to a different horizon. Conversely, a useful decision signal can be concentrated in a small subset that contributes little to an overall score. After evaluating the forecast, assess the pre-specified decision rule separately on dates that did not select it, using realistic net returns and uncertainty estimates.

An adequate report lets another researcher reproduce the event, probability, outcome, score formula, convention, reference, sample, and evaluation timing. It should say whether probabilities are out of sample, whether outcomes overlap, how missing cases were handled, and how many alternative models or scoring choices were tried. That discipline makes a forecast score informative without turning it into a claim about future profit.