Source: Ankur Sinha, Hugging Face financial_phrasebank (50agree subset), 4,846 sentences labeled {positive, neutral, negative} by ≥50% annotator agreement. Downloaded free from https://huggingface.co/datasets/financial_phrasebank/.../FinancialPhraseBank-v1.0.zip into data/finphrase/ (gitignored).
Sample: n=4846 (full set, or --n-rows N to subsample). Caveat: this is the training data ProsusAI/finbert was fine-tuned on, so it's an IN-SAMPLE check — overstates accuracy. A truly out-of-sample test would use FiQA-2018, SemEval-2017 task 5 fine-grained, or Twitter financial sentiment.
| Model | Source | Training data | 3-class | Notes |
|---|
| keyword proxy | POS/NEG lists | none (heuristic) | yes | used as production fallback |
| ProsusAI/finbert | ProsusAI/finbert | Financial PhraseBank | yes (3-class) | currently wired in production (production/correlation_tracker.py:62) |
| cardiffnlp/twitter-roberta-base-sentiment-latest | Cardiff NLP | Twitter (general) | yes (3-class) | general-domain, may generalize beyond finance |
| finiteautomata/bertweet-base-sentiment-analysis | UTAustin CSE | Twitter (financial-relevant) | yes (3-class) | Twitter-trained, maps to finance-style headlines |
| yiyanghkust/finbert-tone | broken on HF (Unrecognized model_type) | n/a | n/a | skipped (config error) |
| Scorer | Accuracy | Macro-F1 |
|---|
| all-neutral baseline | 0.594 | 0.248 |
| keyword proxy (POS/NEG lists) | 0.631 | 0.425 |
| ProsusAI/finbert (3-class) | 0.890 | 0.883 |
| cardiffnlp/twitter-roberta (3-class) | 0.675 | 0.575 |
| finiteautomata/bertweet (3-class) | 0.707 | 0.657 |
| Scorer | Direction accuracy | n (both directional) |
|---|
| keyword | 0.689 | 547 |
| ProsusAI/finbert | 0.984 | 1870 |
| cardiffnlp | 0.990 | 701 |
| bertweet | 0.992 | 1035 |
Direction matters more than 3-class for the bot (we skip neutrals in the daily panel anyway), so this is the most decision-relevant number.
| Pair | κ |
|---|
| keyword vs ProsusAI/finbert | 0.176 |
| keyword vs cardiffnlp | 0.105 |
| keyword vs bertweet | 0.121 |
| ProsusAI/finbert vs cardiffnlp | 0.317 |
| ProsusAI/finbert vs bertweet | 0.450 |
| cardiffnlp vs bertweet | 0.615 |
| true \ pred | positive | neutral | negative |
|---|
| positive | 300 | 1012 | 51 |
| neutral | 144 | 2681 | 54 |
| negative | 119 | 408 | 77 |
| true \ pred | positive | neutral | negative |
|---|
| positive | 1255 | 86 | 22 |
| neutral | 287 | 2470 | 122 |
| negative | 7 | 11 | 586 |
| true \ pred | positive | neutral | negative |
|---|
| positive | 468 | 889 | 6 |
| neutral | 248 | 2578 | 53 |
| negative | 1 | 377 | 226 |
| true \ pred | positive | neutral | negative |
|---|
| positive | 719 | 637 | 7 |
| neutral | 413 | 2397 | 69 |
| negative | 1 | 295 | 308 |
- Best on this set: ProsusAI/finbert (0.890 acc).
- All transformer models beat the keyword proxy by a wide margin (0.890 / 0.675 / 0.707 vs 0.631). Keep the FinBERT wiring; the keyword proxy is a useful fallback only if transformers/HF Hub is unreachable.
- In-sample: this dataset trained FinBERT, so the headline accuracy is upward-biased. Real out-of-sample needs FiQA-2018 / SemEval-2017 task 5 / Twitter financial sentiment (also free on HF).
- Domain: Financial PhraseBank is short earnings-press-release style. Our live input (yfinance headlines) is shorter, noisier, and broader (M&A, macro, CEO changes). The lift on this set may overstate the lift on live data.
- Inference speed: full 4,846 takes ~3-5 min on CPU with batch 32. Live bot batches 50-name pilot in one pipeline call; on the order of 1-2s per day once the model is loaded (lazy-load, ~440MB cache after first run).
Per-row CSV (label, keyword, finbert, cardiffnlp, bertweet, correct_*): reports/finbert_per_row.csv