andreadm/reddit-pulse-bert

Reddit-pulse InflaBERT

Retrained checkpoint. This model was trained again after the paper, on the same gold dataset and with the same seed protocol. The metrics on this card come from that retraining and may differ from the ones reported in the paper.

Reddit-pulse InflaBERT is a three-way directional inflation-expectation classifier for short English texts about the economy, fine-tuned on Reddit submission titles. Given a title (or any sentence-length text), it predicts whether the text conveys that inflation / prices are going up, going down, or carries no directional signal (neutral).

It is not a sentiment model: "inflation falls sharply" is good news but labelled down; "rents are out of control" is bad news but labelled up. The direction is about the price level, not the mood.

The checkpoint is the model behind the Reddit inflation signal in

Del Monaco, A., Longo, L., Marcucci, J. & Tafani, I. (2026). Reddit's 'pulse' on US inflation: forecasting with large language models. Journal of Applied Econometrics, forthcoming. Working-paper version: Banca d'Italia, Questioni di Economia e Finanza (Occasional Papers) No. 1028, June 2026, doi:10.32057/0.QEF.2026.1028.

<!-- TODO: add the JAE DOI / article URL once assigned. -->

The fine-tuning and full-corpus inference code lives at andrea-dm/reddit-pulse.

Model lineage

Architecture: RobertaForSequenceClassification, 6 hidden layers, hidden size 768, 12 attention heads, 50 265-token byte-level BPE vocabulary, 512-token context. Every parameter was updated (no adapters, no quantization); the classification head was re-initialised for the three directional classes.

Labels

How to use

from transformers import pipeline

clf = pipeline("text-classification", model="andreadm/reddit-pulse-bert")
clf("Fed officials warn prices will keep climbing as CPI hits 40-year high")
# [{'label': 'up', 'score': 0.999}]

Batched, with the paper's -1 / 0 / +1 encoding:

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

name = "andreadm/reddit-pulse-bert"
tokenizer = AutoTokenizer.from_pretrained(name)
model = AutoModelForSequenceClassification.from_pretrained(name).eval()
encoding = {"down": -1, "neutral": 0, "up": 1}

texts = [
    "Inflation expectations drop to lowest level since 2021, NY Fed survey shows",
    "What is the difference between CPI and PCE?",
]
with torch.no_grad():
    batch = tokenizer(texts, padding=True, truncation=True, max_length=512, return_tensors="pt")
    ids = model(**batch).logits.argmax(dim=-1).tolist()

print([encoding[model.config.id2label[i]] for i in ids])   # [-1, 0]

Tested with transformers 5.16 and PyTorch 2.13; the checkpoint uses the standard safetensors + fast-tokenizer layout and loads on any recent 4.x release as well. Inference is cheap: the full paper corpus (≈ 243 k texts) labels in a few minutes on one A100 at batch size 64.

Intended use and limitations

Intended use. Labelling large volumes of short, informal, English, economy-related texts (Reddit titles and comments, headlines, social media posts) with a directional inflation signal that is then aggregated over time — the paper's use case. The model is a building block for a high-frequency indicator, not a stand-alone oracle.

Limitations.

• —Single predictions are noisy. Held-out accuracy is ≈ 74 %; the value of the model comes from averaging thousands of predictions per period, where idiosyncratic errors wash out. Do not rely on any single label.
• —Domain and register. Trained on r/economy, r/Economics and r/wallstreetbets titles about US inflation from 2008 to 2022. Other countries, other registers (formal reports, central-bank prose) and post-2022 vocabulary are out of distribution.
• —Class imbalance. down is the minority class (16 % of the gold set) and the hardest one (test F1 0.67 vs 0.79 for neutral). The loss was class-balanced during training, but down recall remains lower.
• —Short texts. Fine-tuned on titles (median 11 words, max 52). Long comments are truncated at 512 tokens and were not seen during training.
• —Direction, not stance or sentiment. The model does not say whether the author wants inflation to move, nor whether the news is good or bad; only which way prices are said to be going.
• —No factuality. A confidently worded false claim is labelled by its direction, not its truth.

Training data

The gold set is a hand-labelled sample of 1 383 Reddit submission titles from r/economy, r/Economics and r/wallstreetbets, dated February 2008 to December 2022. Labels were produced by a human-in-the-loop protocol (manual annotation assisted by zero-shot LLaMA-70B, a fine-tuned LLaMA-8B classifier and ChatGPT-assisted adjudication of disagreements) described in the paper. The gold set itself is not redistributed here.

Titles are lower-cased as found on Reddit; median length 11 words (IQR 8–16, max 52).

Training procedure

Seed protocol and model selection

The paper's protocol asks how sensitive the classifier is to which titles it is trained on, so it separates two sources of randomness:

• —19 split seeds (training.seeds) each draw a different stratified 71 / 19 / 10 % train / validation / test partition (981 / 263 / 139 titles) of the same gold set.
• —One fixed seed (42) governs everything else: classification-head initialisation, batch shuffling and dropout. Every split therefore trains the same model the same way on different data.

Each split is fine-tuned independently; the checkpoint with the median test weighted-F1 across the 19 runs (the upper median, rank 10 of 19) is kept and the others discarded. The result is a typical run, not the best one — the reported metrics are an honest estimate of what a rerun yields, not a lucky draw. This checkpoint is split seed 2786505123.

Hyperparameters

The exact TrainingArguments are in `training_args.json` and the governing configuration extract in `training_config.yml`.

Evaluation

Selected checkpoint (split seed 2786505123)

Per class, on the test split:

Confusion matrix (rows = true, columns = predicted):

Most errors are up titles absorbed by neutral (15 of 54): hedged or question-shaped titles whose direction a human infers from context. Confusions between up and down — the ones that would bias an aggregate signal — are rare (8 of 139).

Split sensitivity across the 19 seeds

Test metrics of every run, sorted by weighted F1. The selected checkpoint is the median.

The spread is what a 139-title test set implies (one title ≈ 0.7 accuracy points) and is the reason the paper reports the median run rather than a single split. Full per-seed numbers: `evaluation/`.

Corpus labelling in the paper

Applied to the paper's filtered corpus, this checkpoint labels 33 460 submissions and 209 995 comments (r/economy, r/Economics, r/wallstreetbets, 2008–2022):

Files

Reproducing

git clone https://github.com/andrea-dm/reddit-pulse && cd reddit-pulse
uv venv && uv pip install -e .
reddit run --model inflabert --gpu 0     # 19 seeds, median selection, corpus labelling

The gold set (data/labelled.xlsx) and the subreddit corpus are not part of the repository; see the paper for the data-construction stages.

Citation

@article{delmonaco2026reddit,
  title   = {Reddit's `pulse' on {US} inflation: forecasting with large language models},
  author  = {Del Monaco, Andrea and Longo, Luigi and Marcucci, Juri and Tafani, Irene},
  journal = {Journal of Applied Econometrics},
  year    = {2026},
  note    = {forthcoming},
}

@techreport{delmonaco2026reddit_qef,
  title       = {Reddit's `pulse' on {US} inflation: forecasting with large language models},
  author      = {Del Monaco, Andrea and Longo, Luigi and Marcucci, Juri and Tafani, Irene},
  institution = {Banca d'Italia},
  series      = {Questioni di Economia e Finanza (Occasional Papers)},
  number      = {1028},
  year        = {2026},
  month       = jun,
  doi         = {10.32057/0.QEF.2026.1028},
}

License

Apache License, Version 2.0 (see `LICENSE.md`). The direct parent MAPAi/InflaBERT is MIT-licensed; the deeper lineage (distilroberta-base, mrm8488/distilroberta-finetuned-financial-news-sentiment-analysis) is Apache-2.0, which this checkpoint now matches. The attribution notices these upstream licenses require are included in LICENSE.md.

The views expressed in the paper are those of the authors and do not necessarily reflect those of the Bank of Italy, the Eurosystem, or the European Commission.