"FinBERT vs alternatives + ensemble — backtest on **FiQA-2018 (Pau Riba, test split)**"

Source: FiQA-2018 (Pau Riba, test split). TRUE OUT-OF-SAMPLE — ProsusAI/finbert was NOT trained on this set (disjoint from Financial PhraseBank).

Sample: n=150. Models: keyword proxy (production fallback), ProsusAI/finbert (3-class, wired in production), cardiffnlp/twitter-roberta-base-sentiment-latest, finiteautomata/bertweet-base-sentiment-analysis, and ensemble (mean of all available transformers' signed scores in [-1,1]).

#Headline (3-class accuracy)

#Direction-only accuracy (positive vs negative, both gold and pred directional)

Direction matters more than 3-class for the bot (we skip neutrals in the daily panel anyway), so this is the most decision-relevant number.

#Inter-scorer agreement (Cohen's κ; 1.0 = perfect, 0 = chance)

#Confusion matrices (rows = true, cols = predicted)

#keyword

#finbert

#cardiffnlp

#bertweet

#ensemble

#Verdict

• Best 3-class accuracy: ensemble (0.640).
• Best direction accuracy: bertweet (0.870).
• Ensemble beats the best single transformer by +0.073 on accuracy — keep the ensemble in production.
• Ensemble direction accuracy 0.838 < best single 0.870; ensemble dilutes direction slightly.

#Limitations

• The ensemble is a mean of signed scores, not a learned stacker. It assumes the transformers output comparable units ([-1, 1]); since each has its own scoring curve (FinBERT 3-class confidence, cardiffnlp/bertweet 2-3-class), the mean is a coarse combination. A learned meta-stacker on this data would likely do better, but the gain is small and the mean is a robust default.
• The signed score mapping assumes neutral = 0; transformers vary in how they treat the neutral class. The ensemble will lean more positive/negative than the mean of the three discrete labels would suggest.