thealper2/flan-t5-base-medal-abbreviation-expansion

thealper2/flan-t5-base-medal-abbreviation-expansion

google/flan-t5-base fine-tuned on the MeDAL pre-training subset for context-aware medical abbreviation expansion: given a context window and an abbreviation, the model generates the full expansion.

Task format

Input:

Expand the medical abbreviation based on the context.

Context: {context}

Abbreviation: {abbreviation}

Target: the expansion string only, e.g. transverse aortic constriction.

Usage

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("thealper2/flan-t5-base-medal-abbreviation-expansion")
model = AutoModelForSeq2SeqLM.from_pretrained("thealper2/flan-t5-base-medal-abbreviation-expansion")

context = "the patient was admitted to the ICU because of severe respiratory distress"
abbreviation = "ICU"
prompt = (
    "Expand the medical abbreviation based on the context.\n\n"
    f"Context: {context}\n\n"
    f"Abbreviation: {abbreviation}"
)

inputs = tokenizer(prompt, return_tensors="pt", truncation=True, max_length=320)
outputs = model.generate(**inputs, max_new_tokens=32, num_beams=4)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Training data

MeDAL rows (ABSTRACT_ID, TEXT, LOCATION, LABEL) are converted into (context, abbreviation, target) triples: the abbreviation is the token at word index LOCATION in TEXT, the target is LABEL, and the context is a symmetric word window around LOCATION (MeDAL abstracts are lower-cased and punctuation-free, so there are no sentence boundaries to cut on).

Sub-sampling from the full MeDAL splits uses seeded reservoir sampling; the dataset's own split boundaries are preserved and the test split is used only for the final evaluation.

Training procedure

Evaluation

Test split, deterministic decoding (beam search, num_beams=4, max_new_tokens=32, no sampling).

Ambiguity breakdown

An abbreviation is ambiguous when more than one distinct gold expansion is observed for it in the processed splits.

Context ablation

Two checkpoints trained with identical data, hyper-parameters and seed; the only difference is whether the prompt contains the context.

Limitations

• —Trained on PubMed abstracts (MeDAL), not on clinical notes; distribution shift to real clinical text is untested.
• —MeDAL labels are produced by automatic reverse substitution, so some gold expansions are noisy or non-canonical; metrics are relative to those labels.
• —MeDAL text is lower-cased with punctuation removed; inputs of a different surface form are out of distribution.
• —Generative decoding can produce expansions that are not in the label inventory.
• —Terminology variants ("heart attack" vs. "myocardial infarction") count as errors under exact match.
• —Not a clinical decision-support system and not validated for clinical use.

Citation

@inproceedings{wen-etal-2020-medal,
    title = "{M}e{DAL}: Medical Abbreviation Disambiguation Dataset for Natural Language Understanding Pretraining",
    author = "Wen, Zhi and Lu, Xing Han and Reddy, Siva",
    booktitle = "Proceedings of the 3rd Clinical Natural Language Processing Workshop",
    year = "2020",
    pages = "130--135",
    url = "https://www.aclweb.org/anthology/2020.clinicalnlp-1.15"
}