Francesco-A/BiomedNLP-PubMedBERT-base-uncased-abstract-bc5cdr-ner-LoRA-v1

🧬 BiomedNLP-PubMedBERT-base-uncased-abstract-bc5cdr-ner-LoRA-v1

This is a LoRA fine-tuned adapter based on microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract. It has been fine-tuned on the BC5CDR dataset, which focuses on biomedical named entity recognition (NER) for chemicals and diseases. The adapter is designed to enhance PubMedBERT’s performance on biomedical text-mining tasks by improving its ability to recognize and classify domain-specific entities while maintaining computational efficiency. It achieves the following results on the evaluation set:

• —Loss: 0.1423
• —Precision: 0.8909
• —Recall: 0.9094
• —F1: 0.9000
• —Accuracy: 0.9755

⚠️ Note: This model is shared for educational and testing purposes only. It is not intended for clinical or production use and may not generalize beyond the BC5CDR dataset.

Model overview

• —Base model: PubMedBERT (abstracts) (domain-specific transformer)
• —Task: Named Entity Recognition (token classification)
• —Entities:
• —Chemical
• —Disease
• —O (non-entity tokens)

The model predicts entity spans following the standard BIO tagging scheme, e.g. B-Chemical, I-Disease, O.

Training notebook

The full training and preprocessing pipeline — including dataset preparation, sentence splitting, BIO tagging, and evaluation — is documented in this Kaggle notebook

Example usage

from peft import AutoPeftModelForTokenClassification
from transformers import AutoTokenizer, AutoModelForTokenClassification
from transformers import pipeline

# Define labels
LABEL_NAMES = ["O", "B-Chemical", "I-Chemical", "B-Disease", "I-Disease"]
id2label = {i: label for i, label in enumerate(LABEL_NAMES)}
label2id = {label: i for i, label in enumerate(LABEL_NAMES)}

tokenizer = AutoTokenizer.from_pretrained("Francesco-A/BiomedNLP-PubMedBERT-base-uncased-abstract-bc5cdr-ner-LoRA-v1")
model = AutoPeftModelForTokenClassification.from_pretrained(
            "Francesco-A/BiomedNLP-PubMedBERT-base-uncased-abstract-bc5cdr-ner-LoRA-v1",
            num_labels=len(id2label),
            id2label=id2label,
            label2id=label2id,          
)

ner = pipeline("token-classification", model=model, tokenizer=tokenizer, aggregation_strategy="first")

demo_texts = [
    "Aspirin is often used to treat inflammation, but may cause gastric bleeding.",
    "Naloxone reverses the antihypertensive effect of clonidine.",
]

for t in demo_texts:
    print("\nText:", t)
    preds = ner(t)
    for p in preds:
        print(f'  - {p["word"]}: {p["entity_group"]} (score {p["score"]:.2f})')

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• —learning_rate: 1e-3
• —trainbatchsize: 32
• —evalbatchsize: 32
• —seed: 42
• —optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• —lrschedulertype: linear
• —lrschedulerwarmup_ratio: 0.1
• —num_epochs: 20
• —loadbestmodelatend: True,
• —metricforbest_model: "f1",
• —mixedprecisiontraining: Native AMP

Training results

Framework versions

• —PEFT 0.16.0
• —Transformers 4.45.0
• —Pytorch 2.6.0+cu124
• —Datasets 3.6.0
• —Tokenizers 0.20.3