jimnoneill/pubguard-review-classifier

SetFit with BAAI/bge-base-en-v1.5

This is a SetFit model that can be used for Text Classification. This SetFit model uses BAAI/bge-base-en-v1.5 as the Sentence Transformer embedding model. A LogisticRegression instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

1. 1.Fine-tuning a Sentence Transformer with contrastive learning.
2. 2.Training a classification head with features from the fine-tuned Sentence Transformer.

Model Details

Model Description

• —Model Type: SetFit
• —Sentence Transformer body: BAAI/bge-base-en-v1.5
• —Classification head: a LogisticRegression instance
• —Maximum Sequence Length: 512 tokens
• —Number of Classes: 5 classes <!-- - Training Dataset: Unknown --> <!-- - Language: Unknown --> <!-- - License: Unknown -->

Model Sources

• —Repository: SetFit on GitHub
• —Paper: Efficient Few-Shot Learning Without Prompts
• —Blogpost: SetFit: Efficient Few-Shot Learning Without Prompts

Model Labels

Evaluation

Metrics

Uses

Direct Use for Inference

First install the SetFit library:

pip install setfit

Then you can load this model and run inference.

from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("setfit_model_id")
# Run inference
preds = model("Vegetables play an importance role in balance diet by providing not only energy but also supplying vital protective nutrients like mineral and vitamins. In addition to their role in nutrition, vegetables increase attractiveness and palatability of a diet by providing sensory appeal through their test and flavors. Vegetables are major and very important constitute of human diet (Thamburaj and Singh, 2005).")

<!--

Downstream Use

List how someone could finetune this model on their own dataset. -->

<!--

Out-of-Scope Use

List how the model may foreseeably be misused and address what users ought not to do with the model. -->

<!--

Bias, Risks and Limitations

What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model. -->

<!--

Recommendations

What are recommendations with respect to the foreseeable issues? For example, filtering explicit content. -->

Training Details

Training Set Metrics

Training Hyperparameters

• —batch_size: (64, 64)
• —num_epochs: (2, 3)
• —max_steps: -1
• —sampling_strategy: oversampling
• —num_iterations: 20
• —bodylearningrate: (2e-05, 1e-05)
• —headlearningrate: 0.01
• —loss: CosineSimilarityLoss
• —distancemetric: cosinedistance
• —margin: 0.25
• —endtoend: False
• —use_amp: False
• —warmup_proportion: 0.1
• —l2_weight: 0.01
• —seed: 42
• —evalmaxsteps: -1
• —loadbestmodelatend: False

Training Results

Framework Versions

• —Python: 3.12.3
• —SetFit: 1.1.1
• —Sentence Transformers: 3.4.1
• —Transformers: 4.57.1
• —PyTorch: 2.8.0+cu128
• —Datasets: 3.4.1
• —Tokenizers: 0.22.2

Citation

BibTeX

@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}

<!--

Glossary

Clearly define terms in order to be accessible across audiences. -->

<!--

Model Card Authors

Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction. -->

<!--

Model Card Contact

Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors. -->

Usage

from setfit import SetFitModel

model = SetFitModel.from_pretrained("jimnoneill/pubguard-review-classifier")
print(model.predict(["<paper abstract here>"]))

Labels: literature_review, research_paper, status_report, survey_study, tool_paper.

If from_pretrained raises TypeError: ... not 'NoneType'

Snapshots of this repo published before 2026-08 were missing the _name_or_path key in config.json. SetFit's model_card.infer_st_id() reads that key and calls Path() on the result, so it raised TypeError: argument should be a str or an os.PathLike object ... not 'NoneType'. Fixed here by adding the key. If you are pinned to an older revision, load the two parts directly:

import json, glob, joblib
from sentence_transformers import SentenceTransformer

d      = "jimnoneill/pubguard-review-classifier"   # or a local snapshot path
labels = ["literature_review", "research_paper", "status_report", "survey_study", "tool_paper"]
body   = SentenceTransformer(d)
head   = joblib.load("model_head.pkl")             # NOTE: joblib, not pickle.load
emb    = body.encode(texts, normalize_embeddings=False)
preds  = [labels[i] for i in head.predict(emb)]

Two gotchas worth knowing: the head is a scikit-learn LogisticRegression written by joblib, so plain pickle.load fails with invalid load key; and it was pickled under scikit-learn 1.6.1, so newer versions emit InconsistentVersionWarning (verified to still produce identical embeddings and predictions on 1.9.0).

Input length

Classify a full abstract. Single-sentence inputs are unreliable and tend to collapse to status_report — verified on inputs whose class was known from full abstracts.

Measured limitations (independent evaluation, 2026-08-04)

Evaluated against NCBI MEDLINE publication types (NLM human indexers) on 1,857 sampled + 4,479 whole-population records from a biomedical corpus. These are field-measured numbers, not training metrics — read them before using this model to filter a training set.

Per-class composition — three of the five classes are mostly primary research

n = 2,424 abstracts with MEDLINE ground truth:

Only `literature_review` identifies reviews. Treating the other classes as "non-research" is expensive: removing survey_study costs 835 research papers to catch 12 reviews (70:1), tool_paper 166 for 5 (33:1), status_report 403 for 99 (4:1).

What `status_report` actually is: its MEDLINE pubtype composition is Journal Article 502 / Case Reports 132 / Review 96 / RCT 25 / Clinical Trial 29. It is a clinical case-report-and-trial class, not a review class.

Recommended operating point

The sum has higher AUC (0.924 vs 0.855), but at a fixed threshold it triples the flag count and halves precision. If your loss is asymmetric — deleting a real paper is worse than keeping a review — use `p(literature_review)` alone.

Era bias — the largest failure mode

Single corpus, abstracts only, no title rules:

Pre-1990 papers are flagged at ~8.7× their true non-research rate, and 9 in 10 of those flags are wrong. The driver is abstract length: shortest quartile (<1089 chars) 43.1% flagged at 34.1% precision; longest quartile 16.5% flagged at 59.0% precision. Older abstracts are short and discursive, which reads as a review. Apply a year guard, or expect to strip the oldest primary literature preferentially.

Venue bias

General-medicine journals over-flag badly; specialist molecular journals are calibrated.

Input requirements

• —Classify an ABSTRACT. In a paired design over 2,424 identical papers scored twice, 179 were flagged on full text but kept on their own abstract — 96.1% of those were primary research. The reverse cell was 4.5× less wrong. Full text degrades precision 47.4% → 39.0%.
• —Not single sentences. Short inputs collapse to status_report.

Domain

Trained on biomedical abstracts. On quantitative/statistical literature the labels become semantically incoherent — Neyman 1923 is tagged status_report, "Line Transect Methods for Plant Surveys" is tagged survey_study on subject matter rather than document type, and the token "model" in a title drives tool_paper to 68% versus a 37.5% baseline. Do not use outside biomedicine without re-validation.

If you need ground truth

MEDLINE publication types are reliable for this task. OpenAlex `type` is not — measured recall against MEDLINE reviews was 1 of 107 (0.9%) on one sample and 1/839 on another; it returns article for essentially everything, including canonical Nature review articles. Two MEDLINE caveats worth knowing: Historical Article and Biography denote historical significance, not document type (NLM applies them retrospectively to landmark primary research), and the retraction pubtype string is Retraction Notice, not Retraction of Publication.