"RL arm diagnostics — what v3 / v4 / meta-allocator actually failed at, and what to try next"

A pre-registered post-mortem on the three REJECTed RL arms in this repo, written before any further RL work. The goal is to map each failure to a specific, testable fix and rank the candidates by expected value.

#1. The three arms and what they actually did

#2. The common failure pattern

The three arms converge on the same diagnosis. The "best" the agent can find on this sleeve set is approximately the equal- weight baseline. Two readings of this:

(a) Pessimistic reading (the prior of the post-mortem author): EW is genuinely hard to beat. The eqmom / gate60 / SPY sleeve set is a set of positively correlated equity-like bets; the agent has no source of decorrelated alpha to exploit. The combined book OOS test (reports/combined_book_oos.md, 2026-08) shows the same thing: tactical sleeves are a drag OOS, passive sleeves are a diversifier, and crypto+PEAD couldn't be OOS-tested at all (in- sample only). An RL allocator looking for marginal alpha in this sleeve set is looking for a signal that isn't there.

(b) Optimistic reading: the RL arms each failed in a specific, diagnosable way that points to a concrete fix. We have not yet exhausted the space; we have exhausted three specific formulations.

The rest of this doc argues (b) is the more productive prior for the NEXT arm, and identifies four candidates. Each candidate addresses a specific, identified failure mode.

#3. The four identified failure modes

#3.1 Cost model was either absent or learned too late

v3 had no cost in env; the agent learned a churn regime and discovered the cost post-hoc. v4 baked cost in and the agent collapsed to EW (no edge). The meta-allocator had no cost and switched 35 times per OOS cell.

The cost-in-env arms (v4) over-corrected: the agent learned "don't move" because the cost gradient was too sharp relative to the expected edge. Specific fix: a soft cost model where the per- unit cost is low at small turnover (matching the maker side on liquid ETFs/crypto) and only steep at large turnover.

#3.2 The action space was too large for the signal

v3 had 4 real-valued weights = 4 degrees of freedom. v4 had 4 Δ-weights = same 4 DoF plus the EW baseline. The meta-allocator had 5 discrete choices = effectively 1 bit of information per day. PPO with 30k timesteps can resolve ~3 bits of information reliably; we're asking it to find a 4-DoF continuous policy in 3.7y of daily data = ~900 obs. That's not enough data for a 4-DoF RL agent. Specific fix: collapse the action space to one DoF (see candidate A: single target-vol allocation).

#3.3 The reward was a Sharpe change, not a Sharpe level

v3's reward was Δ(rolling_60d_Sharpe). This is path-dependent and noisy: a single bar's return can flip the sign of the rolling Sharpe. The Sharpe LEVEL (rolling_60d_Sharpe) is a smoother target. Specific fix: change the reward from Sharpe CHANGE to Sharpe LEVEL, with the DD / turnover / drift penalties preserved.

#3.4 The meta-allocator's switching cost is path-dependent

The meta-allocator's 25 bps/switch is a step function. Two adjacent cells with the same total switching cost (e.g. 25 switches) have the same theoretical net return, but a single cell with 50 switches followed by 76 days of stable rule looks worse than a cell with 25 even switches. Specific fix: make the cost proportional to the per-day turnover fraction, not a binary "did you switch or not".

#4. Four candidates, ranked by expected value

#Candidate A — single-target-vol allocator (highest expected value)

Failure mode addressed: 3.2 (action-space too large) and 3.3 (reward signal too noisy).

The agent's action is a single scalar v_target in [0.05, 0.30] (annualized target vol). The portfolio is then constructed as:

• 50% to the eqmom sleeve
• 30% to gate60 (ETH+BTC basket, gated by the same gate60 signal)
• 20% to SPY (the benchmark, not the bet)
• scale by min(1, v_target / trailing_60d_portfolio_vol) to hit the target vol
• residual is cash

The agent's job reduces to: given the current state, what target vol should I run at? That's 1 DoF, easily trainable, and the answer is grounded in standard portfolio theory (Kelly / vol- target). The reward is Δ(rolling_60d_Sharpe) - λ_DD * max(0, DD - 0.15) - μ * |v_target - v_target_prev| — only one DoF of turnover, so cost is naturally bounded.

The PASS bar is: agent's realized Sharpe > EW Sharpe + 0.10, bootstrap p < 0.05, maxDD < 0.30. Same as the plan.

Why this is different from v3/v4/meta: the action space is 1 DoF, not 4 or 5. The agent's only choice is "how much risk to take" — a problem with a clear theoretical answer (vol-target) and where RL's job is to find the regime-conditional override.

Risk: the agent's realized vol is a function of BOTH the target AND the realized vol of the underlying sleeves. If the sleeves have negative realized vol (the worst regime), the agent can't scale to its target. This is a real risk but it's also exactly the regime-conditional signal we're looking for.

Cost: 1-2 days to implement (env: 50 lines, trainer: reuses rl_v3_trainer.py, OOS + audit + net: reuses rl_v3_oos.py / rl_v3_audit.py / rl_v3_net.py with a 1-DoF adapter).

#Candidate B — conditional overlay on a validated edge (medium-high expected value)

Failure mode addressed: 3.1 (cost model) and 3.2 (action space) AND the meta-issue that the RL arms are looking for marginal alpha in a sleeve set that doesn't have any.

The agent's job is NOT to pick weights. It's to learn WHEN to apply a validated rule. Concretely:

• The PEAD V2 (production/pead_sleeve_v2.py) is VIX-gated. The current gate is VIX > 1y median -> halve size. The agent could learn a richer VIX-conditioned position-sizing rule.
• The carry-basis overlay (production/carry_basis_overlay.py) currently has a 3-bucket rule (carry |z| AND basis |z|). The agent could learn a continuous multiplier in [0.5, 1.5] as a function of (carry z, basis z, funding trend).
• The depth filter (production/depth_filter.py) is binary (skip the bar or don't). The agent could learn a continuous size multiplier as a function of the depth z-score.

This is the "RL as a meta-allocator over a real edge" formulation, which is what the v3 hypothesis was reaching for but never hit because the sleeve set was too abstract.

Why this is different: the agent is now learning a scalar overlay on a rule that already beats the baseline. The signal is in the rule; the agent's job is to find the conditional sizing. This is closer to the carry-basis overlay's actual finding (Sharpe 0.99 → 1.30, p=0.008) — that work was a HAND-CODED conditional rule, not RL. We're asking: can RL find a better conditional rule than the hand-coded one?

Cost: 2-3 days. Need a new env that wraps validation/ carry_basis_overlay.py and exposes (carry_z, basis_z, funding_ trend) as the state and a 1-DoF multiplier as the action.

Risk: the validated edge is small (+0.31 Sharpe over a 0.99 baseline, 252 obs). The agent's job is to find a 5-10% improvement on that, which is below the noise floor on a single walk-forward window. We need at least 10 walk-forward windows of OOS data; the carry-basis study has ~1.5y of OOS data, which is at the edge.

#Candidate C — replace Sharpe CHANGE with Sharpe LEVEL (medium expected value)

Failure mode addressed: 3.3 (reward signal).

The simplest fix of the four. Change models/rl_env_v3.py's reward from Δ(rolling_60d_Sharpe) to rolling_60d_Sharpe itself. The Sharpe level is a smoother signal (doesn't flip sign on a single bar), and PPO's gradient is more stable.

Why this is different: the only change is the reward. Same sleeve set, same action space, same trainer. The training dynamics should be qualitatively different (smoother gradient, better credit assignment).

Cost: 30 min to implement, 1-2 hr to retrain and re-OOS.

Risk: Sharpe LEVEL is a noisy target too (a 60-bar trailing Sharpe has a std error of ~0.4 at daily resolution). The fix might be marginal. But it's a cheap experiment and worth running before investing in candidates A or B.

#Candidate D — explicit hold-period for the meta-allocator (low-medium expected value)

Failure mode addressed: 3.4 (meta-allocator switching cost).

Add hold_period: int to the meta-allocator env (default 5 bars). When the agent switches rules, the new rule is held for hold_period bars regardless of subsequent actions. This bounds the switching cost: 126-day OOS / 5-bar hold = max 25 switches

• 25 bps = 6.25 bps/day = 15% annualized, but in practice ~5 switches * 25 bps / 126 days = 1 bps/day = 2.4% annualized.

Why this is different: the meta-allocator's gross result was mediocre (-0.25 median uplift), but its net was -1.67 because of the switching cost. If the hold period brings the net closer to gross, we can read the gross more clearly.

Cost: 1 hour to implement, 1 hour to retrain.

Risk: the gross was negative too, so this only changes the gap between gross and net. The verdict is still likely REJECT.

#5. Recommendation

Run candidate C first (cheapest, addresses a specific identified failure mode). Then candidate A (highest expected value, also addresses 3.2 and 3.3). Then candidate D (1-hour experiment on the meta-allocator). Save candidate B for last because it's the most expensive and depends on the in-repo carry-basis study having enough OOS data to be statistically meaningful.

If A, C, and D all REJECT, the post-mortem author's pessimistic prior (sleeve set has no marginal alpha) is confirmed. At that point, the right next move is to abandon RL and focus on finding more validated edges (the way carry-basis and depth- filter were found) — the validated-edge pipeline is the one that produces Sharpe 0.5-2.0 improvements, not RL.