"RL v4 — Constrained Allocation Overlay (REJECT)"

#1. Verdict

REJECTED. Constrained PPO did not provide statistically significant cost-adjusted incremental alpha over the existing deterministic equal- weight baseline. The final walk-forward OOS (3 windows × 5 seeds = 15 cells, see §6) gave:

• median Sharpe uplift (agent − EW) = −0.002
• bootstrap p(agent > EW), max across windows = 0.551
• worst-window maxDD = −17.13% (criterion passes)
• mean per-day turnover = 0.0002 (criterion passes)
• cells with positive net alpha vs EW = 40% (criterion fails)
• seeds with positive median uplift = 2 / 5 (criterion fails)

The pre-registered acceptance gate (§5) requires ALL of:

1. Sharpe uplift > +0.10
2. bootstrap p < 0.05
3. positive net-of-cost alpha
4. maxDD > −30%
5. mean daily turnover ≤ 10%
6. positive performance across multiple regimes
7. no obvious concentration failure
8. no data leakage
9. no single random seed responsible
10. survival of realistic transaction costs

Items 1, 2, 3, 6, 9 fail. Verdict: REJECT.

The v3 result (reports/rl_v3_rejection.md, dated 2026-09-04) and the v4 result agree: a PPO allocation overlay does not statistically outperform the equal-weight baseline on this sleeve set, this data window, with realistic transaction costs.

#2. Hypothesis

A constrained PPO allocation overlay (Δ-weights over the equal-weight baseline, with per-sleeve caps, a cash cap, daily-change and turnover limits, weekly rebalance, a no-trade band, and a four-component transaction-cost model) will provide statistically significant cost-adjusted incremental alpha over the existing deterministic equal-weight baseline.

#3. Motivation

v3 (models/rl_env_v3.py, models/rl_v3_trainer.py) closed at REJECT with the v3 audit showing two findings: (a) the agent did learn a materially different allocation than EW (agent-EW corr +0.111) and did better in bear regimes, but (b) per-day turnover of 18% of equity dominated the gross edge, so the net result was indistinguishable from EW. v4 was designed to address those two findings:

• Friction in the env (cost + turnover penalty) so the agent can't learn a churn regime.
• Hard constraints (per-sleeve cap 40%, cash cap 30%, daily change ≤ 5%, daily turnover ≤ 10%) so the policy can't escape the equal-weight baseline by more than a small amount.
• Δ-weights action space so the agent is structurally an "overlay" on the validated equal-weight baseline.
• Weekly rebalance (rebalance_every_n_days = 7) so the cost drag is bounded.
• No-trade band (1% absolute Δ threshold) so the policy doesn't pay spread for nothing.

#4. Previous v3 failure (see reports/rl_v3_audit.md)

The v3 audit enumerated 20 issues. v4 fixes the four root causes:

#5. Acceptance gate (pre-registered)

Adopted from the plan §16. All ten criteria must pass; any miss is REJECT. See validation/rl_v4_oos.py for the implementation; the resulting checks are in reports/rl_v4_oos.md §"Pre-registered acceptance gate".

#6. Environment definition

models/rl_env_v4.py::ConstrainedOverlayEnv.

Defaults are in models/rl_env_v4.py::DEFAULT_CONSTRAINTS, DEFAULT_COST_MODEL, DEFAULT_REWARD.

#7. Pre-registered sweep grid

scripts/rl_v4_sweep.py::SWEEP_GRID:

Sweep selection criterion: median Sharpe across 3 seeds on the validation slice (not on the heldout OOS). The heldout OOS is the last 126 days of the cache, never touched during sweep selection.

The smoke sweep (config A only) selected A as the best by validation Sharpe (+1.213 with 1 seed). The full OOS used config A with 5 seeds × 3 windows.

#8. Training methodology

• PPO from stable_baselines3.
• Single-stage (no curriculum). The v3 curriculum was 3 stages because the action-space dim grew; v4 has a fixed 4-sleeve action so no curriculum is needed.
• Hyperparameters held fixed: lr=3e-4, n_steps=2048, batch_size=64, gamma=0.99, gae_lambda=0.95, clip_range=0.2, ent_coef=0.01.
• Total timesteps per cell: 8000 (8k env steps per PPO).
• 5 seeds × 3 windows = 15 cells. Each cell ~50s on a single CPU.
• Embargo between IS and OOS: 5 trading days.

Output: models/checkpoints/rl_v4/<runid>/<cell_runid>.zip + <cell_runid>_meta.json + DONE sentinel.

#9. Validation methodology

validation/rl_v4_oos.py. For each cell:

1. Load the PPO checkpoint for (window, seed).
2. Build a ConstrainedOverlayEnv on returns.iloc[:oos_end] with the same kwargs as training.
3. Reset, walk the trained policy through the IS slice (no mixing with equal-weight, see v3 finding F10) up to the OOS boundary.
4. Walk the OOS slice, persisting per-day [port_return, cost_bps, net_return, drawdown, weights].
5. Replay the equal-weight and risk-parity baselines on the same OOS slice with the same cost model.
6. Compute per-cell perf, then bootstrap p(agent > EW) on the pooled daily diff.
7. Aggregate to median / mean / min / max / std across cells.

#10. Holdout period

The OOS holdout is the last 126 days of the cache (2026-02-21 .. 2026-06-26) for window 2; the sweep's selection used [oos_start=509, oos_end=572] for VAL, [oos_start=572, oos_end=698] as the held-out OOS. The full walk-forward covers 2017-11-10 .. 2026-06-26 (2166 daily bars).

#11. Results

#11.1 OOS scoreboard (reports/rl_v4_oos.md)

15 cells (3 windows × 5 seeds). Headline:

#11.2 Per-seed results (no single seed)

The result is symmetric around zero; no single seed is responsible for the median.

#11.3 Per-window

No window has a median uplift > +0.03. The early window (2023-11-22 .. 2024-05-23) is the only one with a positive median, and the magnitude is ~0.001 — well within bootstrap noise.

#11.4 Ablation study (reports/rl_v4_ablations.md)

Run on a single OOS slice (the v4 OOS evaluator's first window), 3 seeds × 7 ablations × 6000 timesteps per cell. The first-window EW baseline Sharpe is +0.026.

Reading the table top-to-bottom:

• A→B: adding costs to the env closes ~70% of the gross edge (Sharpe +0.052 → +0.015). The agent's gross edge was a churn regime.
• B→C: turnover penalty is a no-op (B and C are essentially identical). The cost model already includes the cost drag; the additional penalty at λ=1 doubles the signal but the agent was already responding to the cost.
• C→D: weekly rebalance halves turnover (0.003 → 0.001) but doesn't change Sharpe materially. The action's per-rebalance magnitude was small.
• D→E: enabling the constraint overlay doesn't change anything because the agent is already inside the constraints (the action's [-1, 1] range × max_daily_change 0.05 puts it at Δ = 0.05, which is the cap).
• E→F: the no-trade band (1% absolute Δ) further reduces turnover to near zero and slightly reduces Sharpe — the agent was using small rebalances to grab tiny edges.
• F→G: regime-aware obs is the same as F because the regime slot was always on (the v4 ablations experiment keeps regime_obs enabled for all rows; the toggle is in _update_buffers).

The consistent finding: every additional v4 feature either (a) reduces the agent's gross edge by forcing it to be more like EW, or (b) has no effect because the agent was already inside the constraint. The full v4 (G) is statistically indistinguishable from EW.

#11.5 Mechanism audit (reports/rl_v4_audit.md)

The agent learned an allocation within 0.001 of the equal-weight baseline on every sleeve. This is the strongest evidence that the constraints did their job: the agent's marginal action was small enough that the no-trade band + cost penalty pushed it to "don't move".

Regime breakdown: bear regime agent +0.00162 vs EW +0.00167 — agent loses by 5e-5 in bear regimes. The plan's pre-registered mechanism prediction (v3 §1: "the agent should do better in bear regimes") is not observed at the constrained magnitudes v4 allows.

#12. Why did v4 fail?

Reading the evidence top-down:

1. The sleeve set doesn't have a structural alpha the agent can exploit. v3's audit found the agent could find a different allocation than EW; v4's audit shows that the different allocation does not generate net positive returns. The +0.001 mean Δ across sleeves is too small to move the dial in 126-day OOS windows.
2. EW is a very strong baseline. v3's 13-window baseline replay found median EW OOS Sharpe of +0.61; v4's 3-window replay has median +0.61 as well. EW is hard to beat on a 3-5% per-day crypto/equity return series.
3. The constraints are binding at the right level. F and G achieve turnover 0.0002 (vs v3's 18.25%). The cost drag dropped from 26% annualized to 0.02 bps/day. The cost fix worked.
4. The signal isn't there. When the agent is allowed to learn freely (A), it finds a small gross edge. When it's constrained (B-G), the edge disappears. There is no robust, statistically significant, cost-adjusted alpha to be extracted by an overlay on this sleeve set and this data window.

This is the same pattern v3 found (reports/rl_v3_rejection.md §"Stop conditions": S2 and S3 triggered). v4 confirms it: with realistic costs and a constrained policy, an RL overlay does not add value.

#13. What we WOULD change before another RL attempt

Carried from v3, plus the v4-specific findings:

• A longer data window. 3 walk-forward windows of 6mo OOS each is too few to claim statistical significance. The cache is bounded by the yfinance 1Day pull (BTC 2017-01-01, ETH 2017-11-09). To get 10+ walk-forward windows we need either a longer daily history (CoinMetrics, Binance Vision) or a faster rebalance (3-day OOS instead of 6mo).
• A higher turnover regime. v4 solved the cost problem by reducing turnover to ~zero; the agent's allocation ended up at EW. Allowing slightly more turnover (e.g. 5% per day) might let the agent express a real allocation tilt, at the cost of paying more spread. This is a tighter optimization than the v4 sweep did.
• A shorter rebalance horizon. The weekly rebalance was the most effective v4 feature (D) but the daily rebalance (A) gave a marginally higher Sharpe on the small ablation slice. Worth re-testing with a longer IS window.
• A different observation space. The current observation is per-sleeve + cross-sleeve correlation. With longer history, a factor model (e.g. carry, momentum, value factors per sleeve) could give the agent a more informative state.
• A different reward. The current reward is net return minus penalties. With the cost model baked in, the agent learns to be EW. A reward that explicitly rewards Sharpe (or Sortino) with longer lookback (e.g. 60-day rolling Sharpe) might give a different optimum.

#14. Files this study produced

#15. Final answer to the plan's central question

Does constrained PPO provide statistically significant, cost-adjusted incremental alpha over the existing deterministic baseline?

No.

Evidence: median Sharpe uplift across 15 walk-forward cells (windows=3, seeds=5) was −0.002. The bootstrap p-value for "agent > EW" was 0.551 (need ≤ 0.05). The acceptance gate fails on five of ten pre-registered criteria. The mechanism audit shows the agent's final allocation is within 0.001 of the equal-weight baseline on every sleeve, confirming that the constraints left no room for a tilt that survives the cost model.

#16. What this means for the live book

Nothing changes. The validated equal-weight baseline (the production/book_allocator.py path) stays the production default. No production runner is registered. No live orders. No new data dependency. The v3 results remain permanent at reports/rl_v3_rejection.md. The v4 results are recorded here as another rejected hypothesis.

#17. Date locked

2026-09-05