Modelpublic

DT4H/CardioBERTa.it_GP_enriched

sourceHugging Faceupdated 1mo agoView on Hugging Face

0likes10downloads

Model Card

DT4HCardioBERTagrandparentsitenriched

DT4H_CardioBERTa_grandparents_it_enriched is a Italian biomedical terminology encoder for clinical concept normalization and entity linking. It is initialized from [DT4H/CardioBERTa.it] and specialized using CUI-supervised terminology pairs and metric learning.

Backbone

The backbone belongs to the CardioBERTa family from CardioLM - a multilingual suite of small language models for the cardiology domain. CardioBERTa comprises language-specific encoder models adapted to cardiology through continued pretraining on monolingual biomedical and cardiology-related corpora using Masked Language Modeling (MLM). The family covers Czech, Dutch, English, Italian, Romanian, Spanish and Swedish.

Training


Language	Italian (`it`)
Triplet collection	`enriched`
Strategy	`grandparents`
Objective	Multi-Similarity Loss
Mining	All triplets, margin 0.2
Pooling	CLS
Epochs	1
Batch size	256
Learning rate	2e-5
Max. length	25

CUI-supervised terminology pairs enriched with grandparent-level ontology relations.

Terminology statistics

Strategy	Triplets	CUIs	Unique terms	Unique positives	Terms/CUI	Δ terms
synonyms	69,631	69,631	136,720	69,113	2.00	0
parents	1,597,673	476,349	529,199	414,377	3.93	+392,479
grandparents	4,714,271	476,970	529,487	461,831	9.83	+392,767

This model uses 4,714,271 triplets, covering 476,970 CUIs and 529,487 unique normalized terms.

The training terminology is not distributed with this repository because it contains resources subject to UMLS licensing conditions. Only aggregate statistics are released.

Intended use

The model is intended for terminology embedding, biomedical candidate retrieval, concept normalization and entity linking, particularly in cardiology and clinical NLP pipelines. It is not intended for direct clinical decision-making.

Usage

python

import torch
import torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer

model_id = "DT4H/DT4H_CardioBERTa_grandparents_it_enriched"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id)

inputs = tokenizer(
    "clinical concept",
    return_tensors="pt",
    truncation=True,
    max_length=25,
)

with torch.no_grad():
    output = model(**inputs)

embedding = F.normalize(
    output.last_hidden_state[:, 0, :],
    p=2,
    dim=1,
)

Reference

Danu et al. CardioLM - a multilingual suite of small language models for the cardiology domain.

Developed within the DataTools4Heart (DT4H) project, Grant Agreement 101057849.