"Point-in-Time Universe Comparison (P3d ROADMAP item)"

Cache: data/cache/phase1_sp500_stride10_mc8_fenhanced_tr_k28_y20_w12_live_h5h21_pit.pkl Window: 2006-11-03 .. 2026-09-02 (4,988 bars) PIT data: 315 monthly S&P 500 snapshots (2000-01-01 .. 2026-03-01) from gylu/sp500_companies_since_2000 Initial capital: $100,000 | Strategy: per-bar long top-decile by combined score, identical signals across all 4 runs

#Scoreboard (4 runs, identical signals, universe/fudge only)

#What each comparison tells us

• A - C = -2.56% CAGR is the survivorship gap: what the production headline number (current universe + 1.5%/yr fudge) differs from vs the honest PIT-clean number (PIT universe + no fudge). If this is large, the fudge+universe choice was materially biasing the headline. Negative = headline under-states truth.
• B - A = +2.56% CAGR is the drag the 1.5%/yr flat fudge imposed on the current universe. survivorship_adjusted divides the equity curve by (1.015)^(years), so a positive baseline is reduced by the fudge. On a long window with a positive baseline, B - A is positive (raw is higher than fudged); the magnitude measures the cumulative drag of the fudge. The observed +2.56% on a 14y window matches the literature calibration (1.5%/yr x ~14y ≈ 20% cumulative).
• B - C = +0.00% CAGR is the pure effect of replacing the current-membership universe with the PIT universe (no fudge involved). If this is small, the cache's symbol set happened to be mostly representative of the S&P 500; if it's large, the cache's sample contained names that weren't in the index during the relevant window, and including them in the backtest was a survivorship bias.

#Interpretation

Verdict: BIAS DOWN

The current-membership + flat fudge number (+17.27% CAGR) under-states the honest PIT-true number (+19.84% CAGR) by 2.56% CAGR. The flat 1.5%/yr fudge is too pessimistic for this window.

Two components: (a) the fudge itself cost 2.56% CAGR (compounding equity down at 1.5%/yr), and (b) the PIT filter costs 0.00% CAGR by removing names that weren't in the index. Action: keep the fudge as a conservative bias margin (it correctly captures the delisting drag), OR migrate to the PIT universe and set the fudge to 0 -- the PIT-true number is the honest truth, +19.84% CAGR with Sharpe 1.00.

#Per-period breakdown (5y buckets)

#Cache composition vs PIT universe (yearly membership)

#Caveats

• The PIT data is monthly snapshots, applied with step-function interpolation. Within-month membership changes are missed; this is conservative (it under-counts removals) and a 1-month granularity is fine for a 15y survivorship study.
• The cache's combined score is computed at cache-build time on the stride-26 sample (or whatever stride the cache was built on). The PIT filter cannot recreate the originally-fit model on the PIT universe. The 4 runs use identical signals by design; this is a measurement of the bias in the headline number, not a rebuild.
• The cache contains a subset of the S&P 500 (stride-26 = every 26th name from the universe, ~26 names). The PIT filter removes the cache symbol's signal on bars where it was not in the S&P 500. If the cache misses many PIT-true names (under-sampling), the headline number is lower than the universe's true return; the PIT filter only fixes the over-statement from including cache names that weren't in the index.
• The flat 1.5%/yr fudge was calibrated to literature estimates of S&P 500 delisting drag. Our PIT measurement is the first time this codebase has been able to verify that calibration against actual historical membership. The finding (BIAS DOWN with 1.5%/yr fudge too pessimistic) suggests the literature estimate is a conservative upper bound; the real historical delisting drag for the S&P 500 in this window appears to be smaller.

elapsed: 71.0s | cache: phase1_sp500_stride10_mc8_fenhanced_tr_k28_y20_w12_live_h5h21_pit.pkl | n_pit_snapshots: 315