"Phase 1 — Alpaca ETH/USD + SOL/USD Market-Data Ingestion"

Status: DONE | Date: 2026-08-15 | Target branch of the 57-section spec: src/data/alpaca/

#What this phase delivers

A production-grade ingestion layer for the two Phase-1 crypto assets (ETH/USD, SOL/USD, Alpaca spot) covering:

• REST historical backfill (bars, trades, quotes) with sliding-start pagination
• Live websocket collector (trades/quotes/orderbook) with buffer+flush sinks
• Deep order-book snapshot capture (Alpaca's only real depth source)
• Canonical pyarrow schemas, UTC-day partitioning, dedup, DuckDB research views
• Dynamic asset precision (fractionable / min_order_size / increments — no hard-coding)
• Offline test suite (no network) + opt-in live tests
• Documented feed limitations (this page)

#Layout

Spec's src/data/alpaca/ is mapped to the repo's existing data/ convention:

data/raw/alpaca/{SYM}/trades/{date}.parquet             # T
data/raw/alpaca/{SYM}/quotes/{date}.parquet             # Q
data/raw/alpaca/{SYM}/orderbook/{date}.parquet          # O top-of-book rows
data/raw/alpaca/{SYM}/book_snapshots/{date}.parquet     # O full-depth snapshots
data/raw/alpaca/{SYM}/bars/{1Min,5Min}/{date}.parquet   # I (aggregated)

data/alpaca/ modules mirror the spec's src/data/alpaca/ names: common.py, trades.py, quotes.py, orderbook.py, historical.py, websocket.py, market_data.py.

#Files created / modified

#Tests

241 passed, 2 skipped full suite. Phase-1 file: 15 passed, 2 skipped (the 2 skips are the live-gated REST smoke tests; run with RUN_LIVE_TESTS=1). Covered: symbol canonicalization (ETH/USD ok, ETHUSD / ETH-USD rejected), schema round-trips, dedup on dual-write, partitioning across UTC days + end-day boundary, long-form deep book levels, WS reset→snapshot persistence, DuckDB views + bar timeframe extraction.

#Measured feed profile (live, 2026-08-12 → 08-15)

Stored raw rows (3-day backfill + WS soaks):

• ETH/USD fractionable=True, min_order_size=0.000531067, min_trade_increment=1e-9, price_increment=1e-9 (verified live).
• 1Min/5Min bars available back to 2021-01-01 for both symbols; ETH trades back to 2021, but SOL trades/quotes effectively start ~Oct 2024.
• Avg observed spread: ETH ~11 bps, SOL ~45 bps.
• Latest (2026-08-15): ETH ~1,882, SOL ~75.4.

#Feed limitations (explicit, not assumed away)

1. Live websocket is essentially silent. Raw probe against wss://stream.data.alpaca.markets/v1beta3/crypto/us (auth + subscribe + 60s dump): an initial full-depth orderbook snapshot per symbol, then zero trade and zero quote frames for 30s+. The 3-day stored tape confirms the sparse but real tick flow (~275 trades/day). Practical consequence: real-time tick-flow features from WS are NOT available on this feed today; the REST historical + latest-quote endpoints are the usable live path. Initial deep books are captured (book_snapshots) so depth is not lost.
2. No historical deep book. CryptoLatestOrderbookRequest returns only a current snapshot (~60/62 levels/side); no deep-book history. Any depth feature must start accumulating from book_snapshots going forward.
3. SOL trade/quote history starts ~Oct 2024 despite bar history to 2021. Any pre-Oct-2024 SOL feature work must use bars/quotes-only or ETH.
4. Symbols: Alpaca crypto is marginable=False, shortable=False — system constrains to long/flat (never short, no leverage), consistent with spec.
5. Deep book forward-only: partial half-asks / condition confirm reset from the r flag; persistence is append-only per day.

#Decisions locked in

• Canonical symbols strictly ETH/USD, SOL/USD.
• Naive-UTC timestamps; UTC-day partitions via timestamp.dt.normalize().
• nextPageToken unsupported in alpaca-py 0.43.5 → sliding-start pagination.
• DuckDB views registered only when a glob matches (empty-glob guard).
• Model fees: maker 15 bps / taker 25 bps (config) — not yet wired into a backtest; spread/slippage/latency modelling deferred to the execution phase.
• Paper trading first; live orders require explicit confirmation flags.

#Next steps

• Phase 2: feature construction on bars + sparse quotes (microprice/spread proxies), given the WS tick constraint measured here.
• Start accumulating book_snapshots depth history (needs a persistent collector, e.g. periodic reconnect so reset snapshots fire).
• If real-time trade/quotes ever required, evaluate a secondary venue or re-check Alpaca feed upgrades; document the decision.
• Wire fee model + paper-order layer (execution phase).