ISAAC-corpus/isaac-generalization

ISAAC generalization — situation-entity classifier

Same weights as the public DiSCo release. This repository is the ISAAC-facing copy of `BabakScrapes/disco-se-classifier`; the checkpoints are identical. That card carries the full model description and the DiSCo corpus details. This one documents how the model is used inside the ISAAC pipeline and restates the performance figures reported in the ISAAC manuscript. Cite whichever matches your use; if you are using ISAAC labels, cite both.

Clause-level classifier that assigns one of 18 situation-entity types to an English clause, extending Smith's (2003) discourse-mode framework with boundedness distinctions. In the Illinois Social Attitudes Aggregate Corpus (ISAAC) the 18 types are collapsed into three linguistic features that together capture how generalized a statement is:

• —Genericity — is the main referent a generic category (gay people) or a specific individual (my neighbor)?
• —Eventivity — a stable state (God is benevolent) or a transient event (I went to Nebraska)?
• —Boundedness / habituality — for eventive clauses, temporally bounded (I ate this morning), unbounded (God loves us), or habitually recurring (I went there for years)?

The most anecdotal content is specific entities in bounded, non-habitual events; the most generalized is generic categories in stative or habitual clauses. This distinction matters for research on the persuasiveness of attitude-relevant communication, and no other corpus of social group discourse carries it.

Run this model after the clause segmenter, which splits text into the clauses this model scores.

Labels

The released config.json in this repository maps indices to readable names. The full mapping, and the three-feature decomposition ISAAC uses:

QUESTION, IMPERATIVE, NONSENSE, and OTHER are non-statements, for which the three features are undefined. ISAAC counts them separately rather than folding them into the proportions.

Usage

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

REPO = "ISAAC-corpus/isaac-generalization"
tokenizer = AutoTokenizer.from_pretrained("roberta-base", use_fast=True,
                                          add_prefix_space=True)
model = AutoModelForSequenceClassification.from_pretrained(REPO).eval()

clauses = ["Gay people are nice", "my gay neighbor watered my plants"]
enc = tokenizer(clauses, padding="max_length", truncation=True, max_length=128,
                return_tensors="pt")
with torch.no_grad():
    pred_ids = model(**enc).logits.argmax(dim=-1).tolist()

for clause, idx in zip(clauses, pred_ids):
    print(model.config.id2label[idx], "|", clause)

For the full text-to-features pipeline — segmentation, batching, long-text splitting, and per-text aggregation — see code/label_generalization.py in the project repository or generalization.py in the Space.

Training data

Pre-trained on the SitEnt corpus, then fine-tuned on the DiSCo corpus of opinionated, mixed-register English text (Hemmatian, 2022). See the DiSCo card for the corpus description.

Evaluation

Two FacebookAI/roberta-base models run in sequence: a clause segmenter, then a 18-way situation-entity classifier. Both are the same weights as the public DiSCo release — see the clause segmenter and situation-entity classifier cards for the full model description and the corpus they were trained on.

Segmentation. The segmenter covered most of the target clause span for 95.5% of human-verified clauses.

Classification, on the held-out 10% of the disco gold corpus (k ≈ 2,357):

The 18-way macro F1 of .514 is held down by heavy label imbalance across the rarer situation-entity types. The three collapsed features are what ISAAC actually reports, and they are the numbers to rely on.

Intended use

Clause-level annotation of English text for genericity, eventivity, and boundedness/habituality, in aggregate research designs — the role it plays in ISAAC, where it labels clauses across 527M posts.

Out-of-scope use

• —Clause-level, not document-level. Feed it clauses from the segmenter, not whole posts. Scoring an unsegmented paragraph will not give a meaningful label.
• —Not a measure of truth, quality, or bias. A generic statement is not thereby a stereotype, and a specific one is not thereby accurate.
• —Not for individual-level decisions. The 18-way macro F1 of .514 is far too low for any per-clause consequential use; the collapsed features are intended for aggregate proportions over large samples.

Limitations and bias

• —Heavy label imbalance. The rarer situation-entity types are poorly represented in training, which is why 18-way accuracy (.737) far exceeds 18-way macro F1 (.514). Per-class performance on rare types is correspondingly weak. Use the collapsed features.
• —Single-topic fine-tuning. DiSCo is opinionated, mixed-register text on one controversial policy topic. Cross-genre transfer is expected given the formal, content-agnostic nature of the target features, but it has not been separately quantified for Reddit discourse about social groups.
• —English only.
• —Errors compound. ISAAC's generalization columns are the product of two models in series; segmentation errors propagate into classification.

Links

Citation

Please cite the ISAAC paper. One citation covers the whole project — the corpus, the pipeline, and every model. Please do not cite this model repository separately; keeping references in one place is what allows the project's citations to be found together.

@article{hemmatian2026isaac,
  author = {Hemmatian, Babak and Hadjarab, Sarah and Chen, Jessica and Kurdi, Benedek},
  title  = {The {Illinois} Social Attitudes Aggregate Corpus ({ISAAC}): An Open Tool and Reproducible Pipeline for Analyzing Social Group Discourse at Scale},
  year   = {2026},
  note   = {Manuscript submitted for publication}
}

License

Released under a Creative Commons Attribution 4.0 International License. You may use, share, and adapt these weights, including commercially, provided you give appropriate credit — see Citation above.

The ISAAC corpus itself is governed separately by the project Data Use Agreement.