remehostingservices/finance-news-sentiment-35k

Finance News Sentiment 40k

39,965 English financial news headlines, collected from public Telegram finance news-wire channels, labeled for 3-class sentiment (positive / negative / neutral) and a secondary topic label, by two independent LLM judges from different model families with an arbiter settling disputes.

A FinBERT model fine-tuned on this data reaches test accuracy 0.847 / macro F1 0.810: remehostingservices/finbert-finance-news-sentiment. Code, training scripts and the full labeling pipeline: https://github.com/RemeDegen/finance-news-sentiment. How reliable are these labels? A separate audit of the judge and arbiter decisions behind this dataset: https://github.com/RemeDegen/nlp-judge-audit (see Label reliability audit below).

Versions

The validation and test splits are byte-identical in both versions, so results on the test split stay comparable. Load the old version with load_dataset("remehostingservices/finance-news-sentiment-35k", revision="v1-35k").

Data

v1 was split 80/10/10 stratified by sentiment; the v2 rows were all appended to train. No duplicate texts within or across splits. Please report results on the test split as given, so numbers stay comparable.

Source. 8 public Telegram finance news channels (public @usernames, no private groups), February 2023 – June 2026. A deduplicated pool of 142k messages was sampled to 35,000 rows for v1 (32 dropped as too short or not news). v2 adds 5,000 rows drawn from the remaining pool by active learning: the 3,999 headlines on which the v1 model was least certain plus 1,001 random ones (3 dropped as not news). Row counts in the released dataset:

Two channels make up 95% of the data and ~99% of rows are from the last 12 months (June 2025 – June 2026). The channels are news-wire style accounts that repost headlines from agencies (Reuters, Bloomberg, WSJ, official statements); the texts are their headline formulations, not full articles.

from datasets import load_dataset
ds = load_dataset("remehostingservices/finance-news-sentiment-35k")
ds["train"][0]

How the labels were made

1. 1.Blind packets. Rows were shuffled and cut into packets of 40 containing only an id and the text (no channel, date or engagement).
2. 2.Two judges, two model families. Every packet was labeled independently by Claude Opus 5 and by OpenAI gpt-5.6-sol (via Codex), both following the same written rulebook (`judge_spec.md`). The rulebook frames sentiment as investor impact and adds calibration rules: actions over words, tiny price moves are neutral, commodity supply cuts are positive for the commodity, sanctions imposed are negative, polls are neutral, and so on.
3. 3.Agreement and arbitration. On the v1 rows the judges agreed on 85.4% (profile + sentiment + reason). The remaining 5,158 rows were settled one by one by a stronger model, Claude Fable 5, acting as arbiter, seeing both verdicts and the rulebook; the arbiter sided with Opus 5 ~70% of the time, Codex ~27%, and chose a third label ~3%.
4. 4.Audit. Two later blind audits of the v1 labels (300 random rows each, independent Claude Fable 5 sessions with no access to the pipeline) agreed with the final labels 94.3% and 91.3% of the time; 82–85% of the disagreements were on the neutral ↔ directional boundary.
5. 5.v2 round (active learning). The 5,000 added rows went through the same pipeline, rulebook and judges. Because they were picked where the v1 model was uncertain, they are harder than average: the judges agreed on 81.0% (83.5% on sentiment alone). The arbiter settled 1,013 rows: the 952 disputes plus 61 rows where the v1 model contradicted an agreed label with ≥ 0.9 confidence (the judges' label was kept in 59 of those 61). Sides taken: Opus 5 61%, Codex 36%, third label 4%. The label mix of the new rows (57% neutral / 24% negative / 19% positive) is close to v1, i.e. the rulebook was applied consistently across rounds.

The full pipeline (packetizer, judge drivers, collector, arbiter tooling, dataset builder) is open source at https://github.com/RemeDegen/finance-news-sentiment.

Label reliability audit (September 2026)

The judge and arbiter decisions behind this dataset were audited in a separate study, [Two Blind Judges, One Arbiter: A Reliability Audit of LLM Labeling Decisions in a Production Pipeline](https://github.com/RemeDegen/nlp-judge-audit) (report in English and Turkish, analysis code, and row-level tables with both judges' labels and confidence, the arbiter's verdict and which judge it sided with, for every headline in this dataset).

What it found on the 34,998 v1 rows:

• —Disagreement is a threshold problem. The judges disagree on sentiment for 11% of rows; 94% of those disputes are neutral ↔ directional, almost none are positive ↔ negative.
• —Confidence predicts disputes, text does not. The judges' stated confidence predicts which rows will be disputed far better than any text feature (AUC 0.84 vs 0.57). When both judges say "high" (41% of rows) the dispute rate is 0.4%.
• —The arbiter's ~70% lean toward Opus 5 belongs to the production session, not to model family or to the information it saw. Re-arbitrating 595 disputes under eight conditions (three model families, blind and fully informed, fresh sessions), a fresh session of the same model with the same information sided with Opus only slightly above chance, and fresh arbiters of every family defaulted to neutral on 82–96% of disputes.
• —Model error maps label ambiguity. The fine-tuned FinBERT's test error is 3.4% on rows both judges agreed on with high confidence, 19.0% on other agreed rows and 42.9% on arbitrated rows; 82% of the arbitrated-row errors equal the losing judge's label.

Practical consequence: accuracy on this dataset is bounded by how sharply the rulebook resolves the neutral boundary, not by data volume. If you evaluate on the test split, report the arbitrated rows separately; the audit's release tables identify them.

Limitations

• —Labels are LLM-generated. No human gold set exists. Inter-judge agreement (85% on v1 rows, 81% on the harder v2 rows) and the blind audits (91–94%) bound the label quality; a model trained on this data should not be expected to exceed roughly 0.85 accuracy on the test split. The v2 rows moved the fine-tuned model by only +0.5 accuracy, within noise: the label ceiling, not data volume, is the binding constraint. A per-row breakdown of where the labels are uncertain is in the reliability audit above.
• —Neutral is broad. Routine data prints, political statements and small price moves are neutral by design; the threshold for "small" is not perfectly consistent.
• —Source and time concentration. One channel dominates and most rows are from a single year heavy on geopolitics (Iran / Hormuz), tariffs and Fed policy. Expect a drop on other sources or periods.
• —Selection bias in v2. The added rows are not a random sample of the pool; they over-represent boundary cases (neutral vs. directional) by construction.

License

The annotations (sentiment, reason, and everything produced by the labeling pipeline) are released under CC BY-NC 4.0. The headline texts are reproduced from public Telegram channels for research purposes and remain the property of their original sources; no license is granted or implied on them. Check the channels' terms before any commercial use.

Citation

@misc{finance-news-sentiment-40k,
  title  = {Finance News Sentiment 40k: LLM-labeled financial news headlines},
  author = {remehostingservices},
  year   = {2026},
  url    = {https://huggingface.co/datasets/remehostingservices/finance-news-sentiment-35k}
}