"Placebo audit — the edge-discovery \"CANDIDATE\" class, re-tested"

Synthesis review, 2026-09-22. Sources: the individual reports cited below (regenerated this session). No new backtests here; this compiles their verdicts and draws the program-level conclusion.

#Why this audit exists

The 2026-08 edge-discovery program (reports/edge_discovery_batch1..3.md) graded candidates on significance vs zero, regime consistency, cost ladders and DSR. It never tested a candidate against a permutation / random null — the null that asks "would random selection with the same budget do just as well?". That is the missing control, and it is where the near-misses were hiding.

This audit re-ran each surviving candidate through one or more of:

• Score permutation — shuffle the signal within each date; if the ranked selection does not beat random names from the same eligible set, the ranking carries no information.
• Window permutation — same scores, random event windows.
• Random-skip / random-entry — for gate/skip strategies, skip (or enter) the same number of days at random.
• Frequency-matched momentum control — same trade count from a plain momentum score; if it matches, the "signal" is a momentum proxy.

#Results

#Per-edge notes

• H5 looked best on paper (IS=OOS, lag2-robust, 9/11 years) but its Sharpe is below the permutation-null mean; the compression gate actively hurts (PLAIN breakout 0.78 vs H5 0.65), and it does not beat momentum (p=0.861). A textbook momentum mirage.
• H2 is decorrelated from momentum (0.36) but its ranking is noise (p=0.417); the shock+calm gate is worse than plain reversal (0.53 vs 0.78).
• H6 is genuinely informative vs permutation (p=0.000) but 0.95-correlated with raw momentum and does not beat it (p=0.156) — momentum repackaged.
• H10 is the only candidate that survives both the score and window nulls (random windows collapse to 0.49). It is decorrelated from momentum (0.34) with equal Sharpe, so it is a diversifier. But it is 0.72-correlated with the live PEAD sleeve, so it fails the book's diversification gate (h10_diversifier.md: NO-ADD as a 5th sleeve). Correct use = an H10/PEAD blend (blend Sharpe 1.06 > PEAD 0.93 / H10 1.03), a PEAD upgrade, not a new sleeve.
• F7 — the "cleanest near-miss" — is worse than skipping random days (p=0.988). The p=0.156 in the original was the tell.
• R6 beats random entry on average but not significantly (p=0.242), and a frequency-matched 7-day momentum gate scores 1.26 vs R6's 0.55.
• D1 is the closest to surviving (p=0.058) but fails the 0.05 bar; random breakout names on high-volume days already earn 0.58 net2, so most of the "RV edge" is generic high-volume-day beta. D1b's own net10 resid (−0.74) and unpaid-spread caveat agree.

#Generalized gate (whole candidate set)

validation/placebo_gate.py and validation/placebo_gate_events.py turn the ad-hoc batteries into one gate and run it on every equity and event candidate (H1/H7/H8 and PEAD0/H9/H11 were never tested before). Observed = full-window net Sharpe at 10bps; null = within-date score permutation. Size calibration: pure-noise signals are not rejected (0/6 equity, 0/4 event).

Equity (main _ADJ panel):

Events (earnings windows, N=200):

Two notable results: (a) H1/H6/H7/H8 pass the null but are 0.91–1.00 correlated with BASE — real exposures, not new selection edges; (b) PEAD0's own surprise ranking is noise (p=0.395) — the post-earnings "PEAD" return in this panel is the event-window exposure, not the ranking. H10's score-null p is 0.020 (a 40-perm run read 0.000; the stable 200-perm value is 0.020).

F8 DVOL-spike skip (validation/f8_dvol_placebo.py, ETH gate60, random-skip null): skip if eth_z>+1.5 → base Sharpe 1.09 -> 0.89 (lift −0.20) vs null mean −0.02, p95 +0.07, p=0.997 → REJECT — the spike skip is worse than random skipping (it cuts good days). BTC-z REJECT too (p=0.671). The complacency control (drop the 35 most-complacent long days) is a marginal +0.08, p=0.012.

F-family sweep (validation/ffamily_placebo.py, one random-skip harness over the daily crypto gates) — 4 of 5 REJECT: F1 toxicity (lift −0.27, p=0.994), F2 OI-collapse (−0.36, p=1.000), F4b credit (−0.10, p=0.998), F9 stablecoins (−0.02, p=0.942) all skip too much and hurt; only F4 VIX (keep iff VIX <= trailing-252d median) SIGNALs — base 1.09 -> 1.20, lift +0.12, p=0.014. So of the whole F-family, the only gate with information is the equity-VIX macro risk-off filter (marginal); the crypto-microstructure and flow gates are reduced-exposure noise.

#Live sleeves (funded) — placebo check

The gate also validates the funded equity signals (these are not discovery candidates, so a placebo pass is the expected/required baseline):

• EqMom momentum (BASE xsmom126, top decile): placebo p=0.000 — real ranking.
• PEAD live rule (top-quintile surprise per announcement day, 60d hold; production/pead_sleeve.py): OOS Sharpe 0.83 (top-quintile) vs 0.62 (any-positive) vs randomised-quintile null 0.63, placebo p=0.000 (validation/pead_placebo.py) — the surprise-magnitude filter is genuinely informative. NB: this differs from the discovery pead_plain (placebo p=0.395, noise): the live rule ranks within each announcement day and holds a fixed 60d, rather than a rolling-window z top-50.

So the methodology cleanly separates the mirages (H2/H5/F7/R6/D1) from the signals that carry real cross-sectional information (momentum, PEAD quintile, H10's pre-announcement window).

#Crypto core & alt-depth

• ETH depth-filter overlay (validation/depth_filter_placebo.py; live on gate60-ETH, Binance feed): skip the long when z24>+0.5 vs skip the same number at random — base Sharpe 1.05 -> 2.48 (lift +1.43) vs random-skip null mean lift −0.06, p95 +0.09, p=0.000 → SIGNAL. The depth skip is real (and z24>0 even stronger, 2.88). NB: this is the ETH primary; the alt generalization (F6) is REJECT — depth does not transfer to alts.
• CARRY x BASIS sizing overlay (validation/carry_basis_placebo.py; live gate-off): shuffle the {0.5,1.0,1.25} multipliers across gate60-long days vs the carry/basis assignment, OOS>=2025 — 6 of 8 coins SIGNAL (ETH p=0.000, SOL 0.000, AVAX 0.000, DOGE 0.000, ADA 0.001, LINK 0.017), BTC REJECT (p=0.439, inert), XRP borderline (p=0.058). The carry/basis state genuinely decides which long days to size up/down.
• gate60 crypto trend (live core; validation/gate60_placebo.py): random-entry null (same long-day count). OOS>=2025: ETH p=0.052, SOL 0.418, BTC 0.137 — all REJECT (ETH borderline). Full window 2022-12..2026-09: ETH SIGNAL p=0.036, SOL 0.115, BTC 0.095 REJECT. Read: gate60's up-trend timing has modest full-window evidence (ETH) but none in the recent OOS — a yellow flag, not an invalidation (the OOS is short/choppy).
• F6 depth-filter on alts (validation/f6_depth_placebo.py): skipping the gate60-alt long when z24>+0.5 vs skipping the same number at random — lift +0.38 vs random-skip null +0.31, p=0.195 → REJECT. The apparent lift is mostly the benefit of sitting out, not the depth signal (consistent with the F6 study's pooled p=0.603).
• ETF-trend sleeve (validation/etf_trend_placebo.py): the dual-state (close>SMA200 & ROC60>0) selection vs the same number of random basket names (inverse-vol), OOS>=2016 — observed Sharpe 0.77 vs random-entry null mean 0.88, p=0.743 → REJECT. The cross-sectional selection adds nothing over random basket names. (This tests name selection, not the sleeve's de-risking timing.)
• Flagship region-A ML selection (validation/flagship_placebo.py): cross-sectional null (permute combined within each date) — observed OOS Sharpe 1.47 vs null mean 0.79, p95 1.19, p=0.000; temporal null (shift each name's score by ±21/42/63 bars) — the six shifted variants score 0.45–1.40, all below observed, p=0.000. Both the ML score's name assignment and its timing carry real information. Gotcha found and fixed: simulate caches per-symbol row tuples in info["_rowmap"], so any harness that mutates a symbol's df must drop _rowmap or the change is silently ignored (this first presented as a false "BLOCKED" via the sign-flip guard).

#Forward paper (H10 protocol)

validation/forward_paper_report.py scores the observe-only log (logs/edge_discovery_paper.jsonl) once snapshots mature; judge gate >=20 matured. As of 2026-09-23: H10 14 snapshots / 0 matured (log started 09-22), H2/H5/H6 similar → INSUFFICIENT. This is the running forward record for H10; keep research/edge_discovery/paper_tracker scheduled.

#Program conclusion

1. The "CANDIDATE" class is inflated. Seven candidates cleared the program's bars; under a permutation/random null, six are noise or momentum proxies. Significance-vs-zero is not evidence of selection skill on a universe with positive drift.
2. Every "near-miss" (only p fails) was a mirage. F7, R6 and D1 all looked like "one more data point would pass"; all three fail a random null. The recurring pattern: a skip/select filter that mostly captures an exposure (long drift, high-volume beta, momentum) and adds no conditional edge.
3. One survivor: H10. Real, decorrelated from momentum — but redundant with the existing PEAD sleeve, so it is a PEAD complement, not a new edge.
4. Method change (recommended). Add a permutation/random null to the discovery gate before any candidate reaches CANDIDATE. It is cheap and it would have rejected six of seven. Keep significance-vs-zero, but require the placebo bar too.

#Artifacts (this session)

• validation/h5_breakout_holdout.py, validation/h5_breakout_viability.py
• validation/edge_candidates_viability.py
• validation/h10_viability.py, validation/h10_diversifier.py
• validation/f7_fng_placebo.py, validation/r6_funding_ride_placebo.py
• validation/d1_orb_placebo.py
• validation/placebo_gate.py (equity candidates), validation/placebo_gate_events.py
• validation/pead_placebo.py (live PEAD rule)
• validation/f8_dvol_placebo.py (DVOL spike skip)
• validation/ffamily_placebo.py (F1/F2/F4/F4b/F9 batch)
• validation/gate60_placebo.py (crypto core), validation/f6_depth_placebo.py
• validation/depth_filter_placebo.py (ETH depth), validation/carry_basis_placebo.py
• validation/etf_trend_placebo.py, validation/flagship_placebo.py
• validation/forward_paper_report.py (H10 forward harness)
• validation/carry_basis_forward_multicoin.py
• tests: tests/test_h5_breakout_holdout.py, test_edge_candidates_viability.py, test_h10_viability.py, test_h10_diversifier.py, test_f7_fng_placebo.py, test_r6_funding_ride_placebo.py, test_d1_orb_placebo.py, test_placebo_gate.py, test_carry_basis_forward_multicoin.py, test_pead_placebo.py