andreadm/reddit-pulse-llama3.2_3b-xqdora

<!-- Generated by reddit upload. Review before making the repository public: the limitations section is generic, and the per-class breakdown of the hand-written cards is not produced automatically. -->

Reddit-pulse Llama 3.2 3B xQDoRA+

Retrained checkpoint. This model was trained again after the paper, on the same gold dataset and with the same seed protocol. The metrics on this card come from that retraining and may differ from the ones reported in the paper.

Reddit-pulse Llama 3.2 3B xQDoRA+ is a three-way directional inflation-expectation classifier for short English texts about the economy, fine-tuned on Reddit submission titles: a xQDoRA+ adapter on `meta-llama/Llama-3.2-3B` (4-bit NF4 base, DoRA adapters on the attention projections, LoRA+ learning rates). Given a title (or any sentence-length text), it predicts whether the text conveys that inflation / prices are going up, going down, or carries no directional signal (neutral).

It is not a sentiment model: "inflation falls sharply" is good news but labelled down; "rents are out of control" is bad news but labelled up. The direction is about the price level, not the mood.

The checkpoint is one of the small-model classifiers behind the Reddit inflation signal in

Del Monaco, A., Longo, L., Marcucci, J. & Tafani, I. (2026). Reddit's 'pulse' on US inflation: forecasting with large language models. Journal of Applied Econometrics, forthcoming. Working-paper version: Banca d'Italia, Questioni di Economia e Finanza (Occasional Papers) No. 1028, June 2026, doi:10.32057/0.QEF.2026.1028.

The fine-tuning and full-corpus inference code lives at andrea-dm/reddit-pulse.

Model lineage

Labels

How to use

The adapter is loaded on top of the 4-bit quantized base model, exactly as it was trained (bitsandbytes and peft required). config.json in this repository carries the three-way head, the label names and the pad token, so no argument beyond the repository name is needed:

import torch
from peft import PeftModel
from transformers import (
    AutoConfig,
    AutoModelForSequenceClassification,
    AutoTokenizer,
    BitsAndBytesConfig,
)

name = "andreadm/reddit-pulse-llama3.2_3b-xqdora"
tokenizer = AutoTokenizer.from_pretrained(name)
tokenizer.padding_side = "left"
config = AutoConfig.from_pretrained(name)
base = AutoModelForSequenceClassification.from_pretrained(
    "meta-llama/Llama-3.2-3B",
    config=config,
    dtype=torch.bfloat16,
    quantization_config=BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_quant_type="nf4",
        bnb_4bit_use_double_quant=True,
        bnb_4bit_compute_dtype=torch.bfloat16,
    ),
    device_map="auto",
)
model = PeftModel.from_pretrained(base, name).eval()
encoding = {"down": -1, "neutral": 0, "up": 1}

texts = [
    "Inflation expectations drop to lowest level since 2021, NY Fed survey shows",
    "What is the difference between CPI and PCE?",
]
with torch.inference_mode():
    batch = tokenizer(texts, padding=True, truncation=True, max_length=1024, return_tensors="pt")
    ids = model(**batch.to(model.device)).logits.argmax(dim=-1).tolist()

print([encoding[config.id2label[i]] for i in ids])  # [-1, 0]

Do not hand the repository name to AutoModelForSequenceClassification directly: transformers' adapter shortcut rebuilds this DoRA adapter with different logits than the trained model.

Intended use and limitations

Intended use. Labelling large volumes of short, informal, English, economy-related texts (Reddit titles and comments, headlines, social media posts) with a directional inflation signal that is then aggregated over time — the paper's use case. The model is a building block for a high-frequency indicator, not a stand-alone oracle.

Limitations.

• —Single predictions are noisy. The value of the model comes from averaging thousands of predictions per period, where idiosyncratic errors wash out. Do not rely on any single label.
• —Domain and register. Trained on r/economy, r/Economics and r/wallstreetbets titles about US inflation from 2008 to 2022. Other countries, other registers and post-2022 vocabulary are out of distribution.
• —Class imbalance. down is the minority class of the gold set and the hardest one; the loss was class-balanced during training, but down recall remains lower.
• —Short texts. Fine-tuned on titles (median 11 words, max 52). Long comments are truncated and were not seen during training.
• —Direction, not stance or sentiment. The model does not say whether the author wants inflation to move, nor whether the news is good or bad; only which way prices are said to be going.

Training data

The gold set is a hand-labelled sample of 1,383 Reddit submission titles from r/economy, r/Economics and r/wallstreetbets, dated February 2008 to December 2022. Labels were produced by a human-in-the-loop protocol (manual annotation assisted by zero-shot LLaMA-70B, a fine-tuned LLaMA-8B classifier and ChatGPT-assisted adjudication of disagreements) described in the paper. The gold set itself is not redistributed here.

Training procedure

Seed protocol and model selection

The paper's protocol asks how sensitive the classifier is to which titles it is trained on, so it separates two sources of randomness:

• —19 split seeds each draw a different stratified 71 / 19 / 10 % train / validation / test partition (982 / 263 / 138 titles) of the same gold set.
• —One fixed seed governs everything else: adapter and head initialisation, batch shuffling and dropout. Every split therefore trains the same model the same way on different data.

Each split is fine-tuned independently; the checkpoint with the median test weighted-F1 across the runs (the upper median) is kept and the others discarded. The result is a typical run, not the best one. This checkpoint is split seed 2928142788.

Hyperparameters

The exact TrainingArguments are in `training_args.json` and the governing configuration extract in `training_config.yml`.

Evaluation

Selected checkpoint (split seed 2928142788)

Split sensitivity across the 19 seeds

Test metrics of every run, sorted by weighted F1; the selected checkpoint is marked.

The full tables are in `evaluation/`.

Files

Reproducing

git clone https://github.com/andrea-dm/reddit-pulse && cd reddit-pulse
uv venv && uv pip install -e .
reddit run --model llama3.2_3b --gpu 0     # every split seed, median selection, corpus labelling
reddit upload --model llama3.2_3b          # this repository, from the selected checkpoint

The gold set (data/labelled.xlsx) and the subreddit corpus are not part of the repository; see the paper for the data-construction stages.

Citation

If you use this model, please cite the paper it was built for:

@article{delmonaco2026reddit,
  title   = {Reddit's `pulse' on {US} inflation: forecasting with large language models},
  author  = {Del Monaco, Andrea and Longo, Luigi and Marcucci, Juri and Tafani, Irene},
  journal = {Journal of Applied Econometrics},
  year    = {2026},
  note    = {forthcoming},
}

@techreport{delmonaco2026reddit_qef,
  title       = {Reddit's `pulse' on {US} inflation: forecasting with large language models},
  author      = {Del Monaco, Andrea and Longo, Luigi and Marcucci, Juri and Tafani, Irene},
  institution = {Banca d'Italia},
  series      = {Questioni di Economia e Finanza (Occasional Papers)},
  number      = {1028},
  year        = {2026},
  month       = jun,
  doi         = {10.32057/0.QEF.2026.1028},
}

License

The base model is released under the Llama 3.2 Community License, which this adapter inherits (Built with Llama). The upstream notices are included: `LICENSE.txt`, `USE_POLICY.md`.

The views expressed in the paper are those of the authors and do not necessarily reflect those of the Bank of Italy, the Eurosystem, or the European Commission.