Aremaki/LongBEL_1B_SPACCC
LongBEL: Long-Context and Document-Consistent Biomedical Entity Linking
LongBEL
LongBEL is a novel document-level framework for biomedical entity linking (BEL). Instead of normalizing each mention independently, LongBEL conditions each prediction on the document context and on previous normalizations produced in the same document. This design enforces document-level consistency and is enhanced by our robust memory mechanism. The method is introduced in our paper, currently under review.
LongBEL (SPACCC Edition)
This is a finetuned version of LLaMA-3-1B trained on SPACCC, applying the LongBEL framework to enable long context and robust memory predictions.
Intended Use
This model is intended for research on biomedical entity linking and document-level consistency.
It assumes that mention spans and semantic groups are already provided. It does not perform named entity recognition. In a full pipeline, a NER model should first detect mentions and assign semantic groups, then LongBEL can normalize these mentions to SNOMED concepts.
Usage
Loading the model
import torch
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(
"AnonymousARR42/LongBEL_1B_SPACCC",
trust_remote_code=True,
torch_dtype=torch.bfloat16,
device_map="auto",
)Inference example
The model expects BigBio-like documents. Each entity should include a mention text, character offsets, and a semantic group in the type field.
num_beams = 5
bigbio_pages = [
{
"id": "001",
"document_id": "doc_001",
"passages": [
{
"id": "0",
"type": "paragraph",
"text": [
"Una mujer embarazada de 29 años consultó por hipertensión grave, "
"cefalea y dolor epigástrico. Las pruebas de laboratorio mostraron proteinuria. "
"Fue ingresada durante la noche por sospecha de PET y se inició tratamiento urgente."
],
"offsets": [[0, 275]],
}
],
"entities": [
{
"id": "T2",
"type": "ENFERMEDAD",
"text": ["hipertensión grave"],
"offsets": [[45, 63]],
},
{
"id": "T3",
"type": "ENFERMEDAD",
"text": ["proteinuria"],
"offsets": [[131, 142]],
},
{
"id": "T4",
"type": "ENFERMEDAD",
"text": ["PET"],
"offsets": [[239, 242]],
},
],
"events": [],
"coreferences": [],
"relations": [],
}
]
predictions = model.sample(
bigbio_pages=bigbio_pages,
num_beams=num_beams,
)
for i in range(0, len(predictions), num_beams):
mention = predictions[i]["mention"]
print(f"## Mention {(i // num_beams) + 1}: {mention}")
for j in range(num_beams):
pred = predictions[i + j]
print(
f" - Beam {j + 1}:\n"
f" Predicted concept name: {pred['pred_concept_name']}\n"
f" Predicted code: {pred['pred_concept_code']}\n"
f" Beam score: {pred['beam_score']:.3f}\n"
)Example Output:
## Mention 1: hipertensión grave
- Beam 1:
Predicted concept name: hipertensión arterial
Predicted code: 38341003
Beam score: 1.000
- Beam 2:
Predicted concept name: hipertensión arterial sistólica
Predicted code: 56218007
Beam score: 0.024
- Beam 3:
Predicted concept name: hipercaliemia
Predicted code: 14140009
Beam score: 0.003
- Beam 4:
Predicted concept name: hipertrofia de la piel
Predicted code: 24782002
Beam score: 0.001
- Beam 5:
Predicted concept name: hipertensión renal parenquimatosa
Predicted code: 57684003
Beam score: 0.001
## Mention 2: proteinuria
- Beam 1:
Predicted concept name: proteinuria de causa desconocida
Predicted code: 231860006
Beam score: 0.009
- Beam 2:
Predicted concept name: proteinuria no nefrótica aislada
Predicted code: 230970001
Beam score: 0.007
- Beam 3:
Predicted concept name: proteína de la membrana mitocondrial asociada con neurodegeneración
Predicted code: 709415008
Beam score: 0.001
- Beam 4:
Predicted concept name: proteinosis alveolar pulmonar congénita
Predicted code: 707442002
Beam score: 0.000
- Beam 5:
Predicted concept name: proteinosis alveolar pulmonar
Predicted code: 10501004
Beam score: 0.000
## Mention 3: PET
- Beam 1:
Predicted concept name: enfermedad pulmonar obstructiva crónica
Predicted code: 13645005
Beam score: 0.828
- Beam 2:
Predicted concept name: enfermedad pulmonar intersticial
Predicted code: 233703007
Beam score: 0.368
- Beam 3:
Predicted concept name: enfermedad por reflujo gastroesofágico
Predicted code: 235595009
Beam score: 0.334
- Beam 4:
Predicted concept name: petequias cutáneas
Predicted code: 423716004
Beam score: 0.137
- Beam 5:
Predicted concept name: enfermedad pulmonar obstructiva crónica, estadio terminal
Predicted code: 135836000
Beam score: 0.044Saliency map example
The model can also return token-level saliency maps during inference.
predictions, saliency_maps = model.sample(
bigbio_pages=bigbio_pages,
num_beams=num_beams,
with_saliency_maps=True,
)
model.display_saliency_map(saliency_maps[2])Example saliency map for the mention PET:
<p align="center"> <img src="saliency_map.png" alt="Saliency map for PET prediction" width="900"> </p>
Evaluation
Entity linking performance is reported using Recall@1 with bootstrap confidence intervals. The best result is shown in bold, and the second-best result is <u>underlined</u> ⭐ marks the main LongBEL-1B model.
The score reported for this checkpoint is the single LongBEL-1B model. The ensemble result requires fusing several LongBEL input configurations and is not produced by this checkpoint alone.
Speed and Memory
Measured on a single NVIDIA H100 80GB GPU.
LongBEL has the same model memory footprint as the sentence-level Llama-1B baseline, but it is slower because it processes longer contexts and updates document-level memory during inference.
Limitations
This model assumes that mention spans and semantic groups are given. It does not perform mention detection.
LongBEL is most useful when concepts recur within a document. When most concepts appear only once, the memory mechanism has less information to exploit.
Because LongBEL uses previous predictions as memory, early mistakes can still influence later predictions. Robust memory training reduces this risk but does not remove it completely.
This model is intended for research use. It should not be used for clinical decision-making without additional validation and human oversight.
Reproducibility
Code and evaluation scripts are available in this GitHub repository.
Trained model checkpoints and processed datasets are available in the anonymous Hugging Face collection associated with LongBEL.