← all research
STUDY

"Phase 1 — Alpaca ETH/USD + SOL/USD Market-Data Ingestion"

2026-08-15

Status: DONE | Date: 2026-08-15 | Target branch of the 57-section spec: src/data/alpaca/

What this phase delivers

A production-grade ingestion layer for the two Phase-1 crypto assets (ETH/USD, SOL/USD, Alpaca spot) covering:

Layout

Spec's src/data/alpaca/ is mapped to the repo's existing data/ convention:

data/raw/alpaca/{SYM}/trades/{date}.parquet             # T
data/raw/alpaca/{SYM}/quotes/{date}.parquet             # Q
data/raw/alpaca/{SYM}/orderbook/{date}.parquet          # O top-of-book rows
data/raw/alpaca/{SYM}/book_snapshots/{date}.parquet     # O full-depth snapshots
data/raw/alpaca/{SYM}/bars/{1Min,5Min}/{date}.parquet   # I (aggregated)

data/alpaca/ modules mirror the spec's src/data/alpaca/ names: common.py, trades.py, quotes.py, orderbook.py, historical.py, websocket.py, market_data.py.

Files created / modified

FileChangePurpose
data/alpaca/__init__.pynewpackage
data/alpaca/common.pynewsymbol rules, schemas, dedup, PartitionSink, AlpacaDataDB, partition IO
data/alpaca/historical.pynewREST backfill + availability_report() + CLI (python -m data.alpaca.historical backfill|snapshots|report)
data/alpaca/websocket.pynewAlpacaCryptoStream T/Q/O collector (O→top + deep snapshots)
data/alpaca/trades.pynewtrade normalizer + TradesSink
data/alpaca/quotes.pynewquote normalizer + QuotesSink
data/alpaca/orderbook.pyneworderbook (top + long-form deep) rows + sinks
data/alpaca/market_data.pynewMarketDataProvider ABC + AlpacaMarketDataProvider + asset_metadata()
tests/test_alpaca_market_data.pynew15 offline + 2 live tests
pytest.inimodifiedtestpaths + live marker
requirements.txtmodifiedduckdb>=1.0.0
.env.examplemodifiedALPACA_CRYPTO_SYMBOLS, ALPACA_CRYPTO_BAR_TIMEFRAMES

Tests

241 passed, 2 skipped full suite. Phase-1 file: 15 passed, 2 skipped (the 2 skips are the live-gated REST smoke tests; run with RUN_LIVE_TESTS=1). Covered: symbol canonicalization (ETH/USD ok, ETHUSD / ETH-USD rejected), schema round-trips, dedup on dual-write, partitioning across UTC days + end-day boundary, long-form deep book levels, WS reset→snapshot persistence, DuckDB views + bar timeframe extraction.

Measured feed profile (live, 2026-08-12 → 08-15)

Stored raw rows (3-day backfill + WS soaks):

KindRowsDate rangePer-day estimate
trades82308-12→15~275/day (ETH ~200, SOL ~75)
quotes15,61908-12→15~5,200/day (ETH ~3.3k, SOL ~3.0k)
bars 1Min2,87208-12→15both symbols, full coverage
bars 5Min1,33108-12→15both symbols, full coverage
orderbook (top)208-151 WS event / symbol
book_snapshots (deep)19908-15~100 levels / snapshot

Feed limitations (explicit, not assumed away)

  1. Live websocket is essentially silent. Raw probe against wss://stream.data.alpaca.markets/v1beta3/crypto/us (auth + subscribe + 60s dump): an initial full-depth orderbook snapshot per symbol, then zero trade and zero quote frames for 30s+. The 3-day stored tape confirms the sparse but real tick flow (~275 trades/day). Practical consequence: real-time tick-flow features from WS are NOT available on this feed today; the REST historical + latest-quote endpoints are the usable live path. Initial deep books are captured (book_snapshots) so depth is not lost.
  2. No historical deep book. CryptoLatestOrderbookRequest returns only a current snapshot (~60/62 levels/side); no deep-book history. Any depth feature must start accumulating from book_snapshots going forward.
  3. SOL trade/quote history starts ~Oct 2024 despite bar history to 2021. Any pre-Oct-2024 SOL feature work must use bars/quotes-only or ETH.
  4. Symbols: Alpaca crypto is marginable=False, shortable=False — system constrains to long/flat (never short, no leverage), consistent with spec.
  5. Deep book forward-only: partial half-asks / condition confirm reset from the r flag; persistence is append-only per day.

Decisions locked in

Next steps