labbeth/HiT-all-MiniLM-L12-v2-hpo

HPO Hyperbolic Embeddings

A Hierarchy Transformer Encoder (HiT) model fine-tuned to encode Human Phenotype Ontology (HPO) terms in hyperbolic space. The model was developed as part of the Hierarchy-Aware Hyperbolic ans Semantic Reranking workflow for ontology-based phenotype linking from clinical text.

The model is intended to provide a hierarchical signal that complements standard Euclidean semantic retrieval. In the pipeline, it is used for hyperbolic retrieval, hyperbolic reranking, and hybrid reranking of candidate HPO phenotypes.

Important: This model is not a clinical decision-making system. It produces embeddings and distance-based signals that should be used inside a reviewed phenotype-linking workflow. Any medical use requires validation by qualified healthcare professionals.

Model Details

• —Developed by: Thomas Labbé, Moussa Baddour, Axel Bonesteve, Paul Rollier, Marie de Tayrac, Olivier Dameron
• —Model type: Hierarchy Transformer Encoder / hyperbolic text embedding model
• —Base model: sentence-transformers/all-MiniLM-L12-v2
• —Framework: hierarchy-transformers
• —Target ontology: Human Phenotype Ontology (HPO)
• —Training objective: Joint optimization using Hyperbolic Clustering and Hyperbolic Centripetal losses, following the Hierarchy Transformers framework
• —Task: Feature extraction / ontology-aware entity representation
• —Primary downstream use: Candidate retrieval and reranking for HPO phenotype linking
• —License: apache-2.0

Model Sources

• —Paper: Hierarchy-Aware Hyperbolic and Semantic Reranking for Ontology-Based Phenotype Linking
• —Code and data: https://github.com/labbeth/HyperRAG/
• —Framework paper: Yuan He, Zhangdie Yuan, Jiaoyan Chen, Ian Horrocks. Language Models as Hierarchy Encoders. NeurIPS 2024.
• —Hierarchy-Transformers repository: https://github.com/KRR-Oxford/HierarchyTransformers

Intended Use

This model is designed for research and development in biomedical NLP, especially:

1. 1.Encoding HPO phenotype labels and synonyms in a hierarchy-aware embedding space.
2. 2.Computing hyperbolic distances between HPO terms.
3. 3.Reranking candidate phenotypes retrieved from clinical spans.
4. 4.Providing ontology-aware features for phenotype normalization and deep phenotyping workflows.
5. 5.Supporting research on hierarchical ontologies, biomedical entity linking, and retrieval-augmented phenotype extraction.

Out-of-scope use

This model should not be used:

• —as a standalone diagnostic tool;
• —to make autonomous medical decisions;
• —as a replacement for clinician review;
• —for patient-level inference without clinical validation;
• —outside HPO-like hierarchical ontologies without additional evaluation.

Background and Motivation

Phenotype linking requires mapping clinical text spans to structured HPO concepts. Standard dense embedding models often capture local semantic similarity but do not explicitly encode the ontology hierarchy. Hyperbolic geometry is well suited to representing tree-like hierarchical structures, making it useful for ontologies such as HPO.

In the HyperRAG workflow, hyperbolic embeddings are not used as a replacement for semantic retrieval. Instead, they provide a complementary structural signal. The paper reports that hyperbolic retrieval alone does not consistently outperform Euclidean semantic retrieval, but hybrid reranking improves performance and yields more ontology-consistent candidate lists, especially on challenging clinical narratives with implicit phenotype mentions.

Training Data

The model was trained on hierarchical relationships extracted from the Human Phenotype Ontology.

Data construction

Training examples were generated from HPO parent-child relations:

• —positive pairs: direct hierarchical relationships such as <child, parent>;
• —negative pairs: terms that are not parent relationships for the child;
• —synonym augmentation: HPO synonyms were included to improve robustness to alternative phenotype formulations;
• —filtering: synonym combinations were filtered to avoid excessive imbalance.

The resulting training data consists of hierarchy-based triplets derived from the HPO ontology.

Training Procedure

The model follows the Hierarchy Transformers approach. It was fine-tuned from sentence-transformers/all-MiniLM-L12-v2 using hierarchy-aware losses that organize embeddings in hyperbolic space.

Training objectives

The training uses two complementary losses:

• —Hyperbolic Clustering loss: encourages related concepts, especially parent-child concepts, to be close in hyperbolic space.
• —Hyperbolic Centripetal loss: encourages parent concepts to be closer to the origin than their descendants, reflecting hierarchical generality.

Hyperparameters

How to Use

Install the Hierarchy Transformers package following the official repository instructions.

from hierarchy_transformers import HierarchyTransformer

model = HierarchyTransformer.from_pretrained("labbeth/HiT-all-MiniLM-L12-v2-hpo")

terms = [
    "Abnormality of the nervous system",
    "Intellectual disability",
    "Delayed speech and language development",
]

embeddings = model.encode(terms, convert_to_tensor=True)

Compute hyperbolic distances

from hierarchy_transformers import HierarchyTransformer

model = HierarchyTransformer.from_pretrained("labbeth/HiT-all-MiniLM-L12-v2-hpo")

child_terms = ["Delayed speech and language development"]
parent_terms = ["Abnormality of speech or vocalization"]

child_emb = model.encode(child_terms, convert_to_tensor=True)
parent_emb = model.encode(parent_terms, convert_to_tensor=True)

distances = model.manifold.dist(child_emb, parent_emb)
child_norms = model.manifold.dist0(child_emb)
parent_norms = model.manifold.dist0(parent_emb)

# Example hierarchy-aware score inspired by Hierarchy Transformers probing
centri_score_weight = 1.0
subsumption_scores = - (distances + centri_score_weight * (parent_norms - child_norms))

Use inside the HyperRAG hybrid reranker

For a clinical span and a set of candidate HPO terms, HyperRAG combines semantic similarity and normalized hyperbolic distance:

S_hybrid(C_i, span) = gamma * S_cos(C_i, span) - (1 - gamma) * d_H_hat(C_i, span)

where:

• —S_cos is the cosine similarity in Euclidean embedding space;
• —d_H_hat is the normalized hyperbolic distance;
• —gamma controls the balance between semantic and hierarchical signals.

In the paper, gamma = 0.5 is used in the main experiments for simplicity and interpretability.

Evaluation

The model was evaluated both intrinsically and as part of the HyperRAG phenotype-linking workflow.

Intrinsic hierarchy evaluation

The model was compared with the Euclidean base embedding model using distances between:

• —one-hop HPO relations;
• —multi-hop HPO relations;
• —synonyms;
• —negative pairs.

The hyperbolic model showed narrower and lower-distance distributions for hierarchical relations, especially multi-hop relations, indicating better encoding of the HPO hierarchy.

Downstream phenotype-linking evaluation

The model was evaluated in the HyperRAG workflow on:

• —ID-68: a public benchmark for phenotype extraction;
• —CHU-50: a newly released anonymized clinical-note dataset with phenotype annotations, including a substantial proportion of implicit phenotype mentions.

The paper reports that:

• —hyperbolic retrieval alone underperforms strong Euclidean semantic retrieval;
• —hyperbolic embeddings provide complementary ontology-aware signals;
• —hybrid reranking improves weighted recall and ranking quality, especially on CHU-50;
• —hyperbolic and hybrid approaches produce more hierarchically coherent candidate lists.

Why Recall@k Matters

This model is designed for candidate-list generation and reranking, not only Top-1 classification. In deep phenotyping workflows, clinicians often benefit from a coherent list of candidate phenotypes rather than a single binary prediction. A candidate list can support expert review, reduce missed phenotypes, and improve downstream diagnostic reasoning.

Limitations

• —Hyperbolic distance is not always sufficient to capture fine-grained clinical semantics.
• —The model is trained from ontology structure, not from real clinical notes.
• —Hyperbolic retrieval alone may underperform semantic retrieval for exact candidate matching.
• —Performance depends on the structure and quality of HPO.
• —The model may be less appropriate for ontologies with dense transversal relationships rather than mostly hierarchical structure.
• —Clinical deployment requires external validation and human oversight.

Ethical Considerations

The model is intended for research and decision support, not autonomous clinical use. It should be used only in workflows where predictions are reviewed by qualified clinicians.

The CHU-50 evaluation dataset described in the paper is anonymized and was designed to avoid inclusion of personal data. No real clinical data were used for model training.

Community Resources

The HyperRAG project provides the following resources for the community:

• —CHU-50 evaluation dataset in French;
• —CHU-50 evaluation dataset in English;
• —training set for the ColBERT reranker;
• —training set for the HPO hyperbolic model;
• —HPO hyperbolic model weights;
• —code for retrieval, reranking, and hierarchy-aware evaluation.

Citation

If you use this model, please cite both the HyperRAG paper and the Hierarchy Transformers framework.

@inproceedings{labbe2026hyperrag,
  title = {Hierarchy-Aware Hyperbolic and Semantic Reranking for Ontology-Based Phenotype Linking},
  author = {Labbé, Thomas and Baddour, Moussa and Bonesteve, Axel and Rollier, Paul and de Tayrac, Marie and Dameron, Olivier},
  booktitle = {ACL BioNLP 2026 - Biomedical Natural Language Processing Workshop},
  year = {2026}
}

@article{he2024language,
  title = {Language Models as Hierarchy Encoders},
  author = {He, Yuan and Yuan, Zhangdie and Chen, Jiaoyan and Horrocks, Ian},
  journal = {Advances in Neural Information Processing Systems},
  volume = {37},
  pages = {14690--14711},
  year = {2024}
}

Contact

For questions or feedback, please contact the HyperRAG authors through the project repository or the corresponding author listed in the paper.