← all research
REJECTED

"Phase 4 — Per-symbol walk-forward model & OOS evaluation (Alpaca crypto)"

2026-08-15

·

Status: DONE (2026-08-15) · Suite: 267 passed, 2 skipped (+7 tests in tests/test_alpaca_model.py)

What was built

data/alpaca/model.py — rolling walk-forward trainer + honest, fees-aware OOS evaluation for the long/flat bar system. CLI: python -m data.alpaca.model.

Design (repo lessons honored)

  • One model per symbol — never pooled cross-sectional crypto (that failed repeatedly in this repo).
  • GradientBoostingRegressor per symbol with locked defaults: n_estimators=300, max_depth=3, learning_rate=0.05, subsample=0.8, min_samples_leaf=20, random_state=0 (override-able via model_kwargs).
  • Expandable walk-forward: each test block is predicted by a model trained strictly on PRIOR rows minus the embargo (gap_bars = config.ml.embargo_bars, PANEL-style 50-bar default). No randomized/CV splitting anywhere.
  • No forward-looking trains: fitted_until tracks the last training bar; leakage_check() (Phase 3) is invoked per split before predictions are kept.

Guardrails

  • walk_forward_predict: split skipped (no model) when clean training rows < min_train (200) — earlier thin-history blocks become no-trade by design;
  • training decimation: when clean rows > max_train_rows (60k), a stride keeps the newest portion representative without O(n) blowup;
  • predictions NaN on any feature-missing row (never imputed);
  • AlpacaModel.fit refuses < 100 clean rows; inf/NaN rows dropped before fit.

Evaluation (gross stats + explicit fee hurdle)

evaluate_oos on out-of-sample rows only:

  • n, Pearson IC, Spearman, mean y (bps);
  • top/bottom-quintile mean returns + spread + t-stat, gross PF;
  • fee hurdle surfaced, never hidden: maker_rt_bps = 2×fee_maker_pct×1e4 (30 bps round trip), taker_rt_bps = 2×fee_taker_pct×1e4 (50 bps) from config.risk.fee_maker_pct/fee_taker_pct. Economic pass/fail lives in the Phase-5 backtester; this module only reports numbers honestly.

Persistence (models_dir/alpaca/{SYM}/{TF}/)

  • h{h}.pkl — fitted models + metrics + fit timestamp;
  • h{h}_oos.parquet — full OOS [timestamp, y, pred];
  • h{h}.json — summary (IC, top-quintile bps, fee hurdles) for reports/UIs.
  • load_pipeline() for reuse (Phase 9 live inference will load h{h}.pkl).

CLI

python -m data.alpaca.model --symbols ETH/USD,SOL/USD --timeframes 1Min,5Min \
       --h {config.ml.prediction_horizon} --n-splits {walk_forward_splits} \
       --gap {embargo_bars} [--start YYYY-MM-DD] [--end YYYY-MM-DD]

Tests (tests/test_alpaca_model.py, isolated dirs → tmp)

  • fit/predict alignment with NaN-on-gap; fit refuses dirty rows;
  • walk-forward OOS: injected-signal dataset reaches IC ≈ 1 with positive top-quintile means; noise-only dataset gives IC ≈ 0 (sanity floor);
  • early thin splits skipped without breaking the pipeline;
  • run_pipeline persists (pkl/json/parquet) and load_pipeline reads back;
  • evaluate_oos respects configurable maker/taker fees.

Known caveats

  • GBR hyperparameters are sensible defaults, not tuned — Phase 5's backtester is the arbitrating cost model; tuning only after the honest baseline exists.
  • Very early splits often skip on 1Min history (warmup + min_train) — that is intended: thin-history models are worse than no signal.