A Falsification-First Protocol for Retail Strategy Research

Every systematic trader has a graveyard of strategies that "worked" until they didn't. The difference between a hobbyist and a professional is rarely access to better signals — it is the discipline and machinery to kill bad ideas before capital does.

This post documents the protocol I use to research, validate, and (usually) reject trading strategies. It was built iteratively over two years of daily systematic research across crypto and US equities, and it has one defining property: most candidates die. Of roughly twenty distinct edge families tested to completion, four survived. The survivors are modest, honest about being part beta-harvest, and worth trading precisely because everything else was falsified first.

The pipeline

candidate idea
   → full-grid backtest (no cherry-picking, all cells reported)
   → calibration / held-out split (tune only on calibration)
   → Monte-Carlo permutation null + bootstrap
   → cost & execution stress (maker vs taker, slippage ladders)
   → virgin holdout (never touched during development)
   → paper trading → adoption or autopsy

Each stage exists because a specific class of self-deception got through an earlier version of the pipeline. What follows are the stages and the failures that motivated them.

1. Full grids, no selection on the test window

Any parameter sweep is reported in its entirety, including ugly cells. Selection happens only on a calibration window; the held-out half is scored once, at the end, with whatever parameters the calibration chose.

The failure this prevents: the classic "we tried 400 configurations and the best one made 800%" — which is just noise mining with extra steps. In one study, a trend rule applied across 28 ETFs produced 448 grid cells; reporting all of them (340 profitable, 108 not) is what makes the 28/28 headline meaningful rather than suspicious.

2. Walk-forward with embargo

Models retrain on strictly-past data at fixed intervals; predictions apply only after an embargo gap past each training cutoff. A leakage audit runs poison controls — deliberately injecting the true future return as a feature must produce absurdly profitable backtests, proving the harness can detect real signal when it exists:

• Poison feature alone: OOS information coefficient +0.19, backtest +13,000,000%
• Poison among 117 real features: still detectable (+0.137 IC)
• Real features without poison: negative OOS IC at every feature count

That last line closed an entire research direction (ML return prediction) on evidence rather than preference. The harness converts signal into profit; the signal was simply not there.

3. Permutation nulls and bootstrap

Anything that survives the split faces a selection-aware permutation null: shuffle within-day score ranks (preserving eligibility masks), rerun the full backtest 40–1000 times, and ask how often random books beat the realized one. Trade-level bootstrapping gives P(loss) and confidence intervals.

A candidate that passes its tuning gate but sits at p = 0.05 against nulls gets rejected — one did exactly that (p = 0.050, 2 of 40 permutations already beat it), and the ledger says so.

4. Cost stress as a first-class gate

Retail edges die by fees more often than by signal absence. Every backtest reports maker/taker fee ladders, slippage on turnover, and borrow costs for any short leg. Thirteen separate intraday crypto edge families — order-flow imbalance, VPIN, taker-flow, lead-lag, basis, carry, funding, positioning — all showed plausible gross behavior and all died here. An overnight-premium effect earned +8.9%/yr gross and lost −34%/yr after daily round trips.

5. Virgin holdouts

Some data is never touched during development. One intraday equity model looked reasonable in-sample and then scored profit factor 0.75 on its virgin holdout. It was never deployed. Holdout data that gets peeked at stops being holdout.

Case study: the bug that beta-checking caught

During a dollar-neutral long/short study over ~450 stocks, the first backtest returned +467% with a Sharpe of 0.46. Attractive — except a market-neutral book should have near-zero market beta, and this one measured 0.85.

Chasing that inconsistency found a genuine implementation bug: position weights were persisted via replace(0, nan).ffill(), which converts meaningful flat positions into NaN and forward-fills stale weights forever. The result was an accidental ~8.7× gross-levered portfolio wearing a market-neutral costume.

Three controls settled it:

1. Static per-name betas of the long and short legs were both ≈1.03 — neutral
2. A random-rank control book built through identical machinery measured β≈0
3. After fixing persistence, the honest book returned −2%/yr net of costs

The lesson generalizes: when a result looks too good, interrogate the machinery with controls, not the result with more parameters. The fixed study still taught something — a faint gross spread exists (Sharpe ≈ 0.45 gross, negative net) and was rejected on economics, not on disappointment.

The ledger discipline

Every completed test enters a decision ledger with its verdict, numbers, and reason. Recent entries include:

Rejections are recorded with the same care as adoptions. The ledger is the product; individual strategies are just samples from it.

What survived

Four things passed the full gauntlet: a walk-forward ML equity rotation (+27.9% CAGR on its untouched test window), a multi-asset daily trend portfolio (28/28 assets positive), a crypto trend overlay, and a combined allocation frontier tying them together. Their stories — including honest caveats about how much of their return is filtered beta rather than alpha — are in the other posts in this series.

None of this is investment advice, and all live accounts described elsewhere on this site are exchange paper-trading accounts.