"FinBERT vs alternatives + ensemble — backtest on **Financial PhraseBank v1.0 (50agree)**"

Source: Financial PhraseBank v1.0 (50agree). IN-SAMPLE to ProsusAI/finbert (its training data). Use FiQA-2018 for true OOS.

Sample: n=500. Models: keyword proxy (production fallback), ProsusAI/finbert (3-class, wired in production), cardiffnlp/twitter-roberta-base-sentiment-latest, finiteautomata/bertweet-base-sentiment-analysis, and ensemble (mean of all available transformers' signed scores in [-1,1]).

#Headline (3-class accuracy)

#Direction-only accuracy (positive vs negative, both gold and pred directional)

Direction matters more than 3-class for the bot (we skip neutrals in the daily panel anyway), so this is the most decision-relevant number.

#Inter-scorer agreement (Cohen's κ; 1.0 = perfect, 0 = chance)

#Confusion matrices (rows = true, cols = predicted)

#keyword

#finbert

#cardiffnlp

#bertweet

#ensemble

#Verdict

• Best 3-class accuracy: finbert (0.892).
• Best direction accuracy: cardiffnlp (1.000).
• Ensemble does NOT beat the best single transformer on accuracy (-0.076); the mean dilutes the strongest signal. Stick with a single model (FinBERT) in production.
• Ensemble direction accuracy 0.972 < best single 1.000; ensemble dilutes direction slightly.

#Limitations

• The ensemble is a mean of signed scores, not a learned stacker. It assumes the transformers output comparable units ([-1, 1]); since each has its own scoring curve (FinBERT 3-class confidence, cardiffnlp/bertweet 2-3-class), the mean is a coarse combination. A learned meta-stacker on this data would likely do better, but the gain is small and the mean is a robust default.
• The signed score mapping assumes neutral = 0; transformers vary in how they treat the neutral class. The ensemble will lean more positive/negative than the mean of the three discrete labels would suggest.