HiyawErtiro/cryptobert-finetuned_on_news_headlines

Update

Updated the model with more real headlines(2024-July 2026) + 1k more syntethic data.

CryptoBERT Fine-Tuned on Bitcoin News Headlines

A directional sentiment classifier for Bitcoin news headlines, fine-tuned from ElKulako/cryptobert.

Model Description

CryptoBERT was trained on retail social media posts (StockTwits, Twitter, Reddit), which gives it strong crypto-specific vocabulary but a poor match for journalistic headline text. Zero-shot, it scores 0.39 macro-F1 on Bitcoin headlines, close to random for a 3-class task. This model fine-tunes CryptoBERT on ~41,400 headlines relabeled with a directional market-impact rubric, reaching 0.78 macro-F1 on an independent held-out set.

• —Developed by: Hiyaw Ertiro
• —Model type: BERT-based sequence classification (3-class)
• —Language: English
• —License: MIT
• —Finetuned from model: ElKulako/cryptobert

Labels

Labels are directional (does the event plausibly move price), not tonal (does the headline read positive/negative). See the dataset card for the full labeling rubric.

Direct Use

Sentiment scoring of English-language Bitcoin news headlines, as a feature for downstream pipelines (price prediction, trading signal aggregation, market monitoring dashboards).

How to Get Started

from transformers import AutoModelForSequenceClassification, AutoTokenizer
import torch

model = AutoModelForSequenceClassification.from_pretrained("hiyawtaken/cryptobert-finetuned_on_news_headlines")
tokenizer = AutoTokenizer.from_pretrained("hiyawtaken/cryptobert-finetuned_on_news_headlines")

headline = "Bitcoin surges to new all-time high as ETF inflows accelerate"
inputs = tokenizer(headline, return_tensors="pt", truncation=True, max_length=128)
with torch.no_grad():
    logits = model(**inputs).logits
pred = torch.argmax(logits, dim=-1).item()
print(model.config.id2label[pred])

Training Details

Training Data

labeled-bitcoin-news-headlines, built on top of Bitcoin News Sentiment Dataset by filipemunizz (2011-2024) + headlines pulled from google news(2024-July 2026). The original tone-based labels were relabeled with DeepSeek V4 Flash using an explicit directional rubric, with noise removed (ads, listicles, evergreen explainers, non-English headlines, generic roundups). ~41,400 real headlines plus ~500 synthetic headlines (fictional companies, clearly disclosed as synthetic) added to correct two specific model failure patterns found during validation: reasoning about transaction magnitude ($2 vs. $2M) and disambiguating near-identical dollar figures referring to different underlying events.

Training Procedure

Full fine-tune (all parameters updated, not a frozen-encoder head-only tune), warm-started from CryptoBERT's existing 3-class head.

Training Hyperparameters

• —Base model: ElKulako/cryptobert
• —Epochs: 3
• —Batch size: 32
• —Learning rate: Dynamic
• —Training regime: fp16 mixed precision
• —Metric for best checkpoint: macro-F1 (not accuracy, due to class imbalance)

Testing Data

An independent, hand-labeled set of 72 live-scraped 2026 Bitcoin headlines, never seen during labeling or training.

Metrics

Macro-F1 was used instead of accuracy because the classes are imbalanced (neutral is the largest class), and accuracy alone would reward a model that defaults to the majority class.

Results

Summary

Fine-tuning improves macro-F1 by +38.5 points over zero-shot CryptoBERT on headline-length text, confirming that CryptoBERT's social-media training data transfers poorly to journalistic headlines without adaptation.

Bias, Risks, and Limitations

• —Training labels are LLM-generated (DeepSeek V4 Flash), not human-annotated. Treat as strong weak supervision, not ground truth.
• —The neutral/positive boundary is inherently fuzzy for headlines with implied but unstated direction (price predictions, analyst commentary). Some disagreement with human judgment is expected.
• —The model has limited ability to reason about transaction magnitude and to disambiguate near-identical numeric details referring to different underlying events, even after targeted synthetic data augmentation. This is a known limitation of a 110M-parameter text classifier rather than something expected to fully resolve with more data of the same kind.

Recommendations

Use as one signal among several in a larger pipeline, not as a standalone decision-maker. Review model predictions on headlines involving specific dollar amounts or company financial actions with extra scrutiny given the documented magnitude-reasoning limitation.

Model Architecture and Objective

BERT-based (RoBERTa-family) sequence classification, 3-class softmax output.

Hardware

NVIDIA RTX 3090 (24GB VRAM)

Software

PyTorch, Hugging Face Transformers, Datasets

Model Card Contact

Hiyaw Ertiro — https://huggingface.co/datasets/HiyawErtiro/bitcoin-news-sentiments-latest