thellert/accphysbert_cased

AccPhysBERT

AccPhysBERT is a specialized sentence-embedding model fine-tuned for accelerator physics, capturing semantic nuances in this technical domain. It delivers state-of-the-art performance in tasks such as semantic search, citation classification, reviewer matching, and clustering of accelerator-physics literature.

Model Description

• —Architecture: BERT-based, fine-tuned from PhysBERT (cased) using Supervised Contrastive Learning (SimCSE).
• —Optimized For: Titles, abstracts, proposals, and full text from the accelerator-physics community.
• —Notable Features:
• —Trained on 109 k accelerator-physics publications from INSPIRE HEP
• —Leverages 690 k citation pairs and 2 M synthetic query–source pairs
• —Trained via SentenceTransformers to produce dense, semantically rich embeddings

Developed by: Thorsten Hellert, João Montenegro, Marco Venturini, Andrea Pollastro Funded by: US Department of Energy, Lawrence Berkeley National Laboratory Model Type: Sentence embedding (BERT-based, SimCSE fine-tuned) Language: English License: CC BY 4.0 Paper: Domain-specific text embedding model for accelerator physics, Phys. Rev. Accel. Beams 28, 044601 (2025) https://doi.org/10.1103/PhysRevAccelBeams.28.044601

Training Data

• —Core Corpus:
• —109,000 accelerator-physics publications (INSPIRE HEP category: "Accelerators")
• —Over 1 GB of full-text markdown-style text (via OCR/Nougat)

• —Annotation Sources:
• —690,000 citation pairs
• —49 semantic categories labeled via ChatGPT-4o
• —2,000,000 synthetic query–source pairs generated with LLaMA3-70B

Training Procedure

• —Fine-tuning Method: SimCSE (contrastive loss)
• —Hyperparameters:
• —Batch size: 512
• —Learning rate: 2e-4
• —Temperature: 0.05
• —Weight decay: 0.01
• —Optimizer: Adam
• —Epochs: 2
• —Infrastructure: 32 × NVIDIA A100 GPUs @ NERSC
• —Framework: SentenceTransformers

Evaluation Results

AccPhysBERT outperforms BERT, SciBERT, and large general-purpose embedding models in all accelerator-specific benchmarks.

Example Usage

from transformers import AutoTokenizer, AutoModel
import torch

tokenizer = AutoTokenizer.from_pretrained("thellert/accphysbert")
model = AutoModel.from_pretrained("thellert/accphysbert")

text = "We report on beam instabilities observed in the LCLS-II injector."
inputs = tokenizer(text, return_tensors="pt")
outputs = model(**inputs)

# Use mean pooling (excluding [CLS] and [SEP])
token_embeddings = outputs.last_hidden_state[:, 1:-1, :]
sentence_embedding = token_embeddings.mean(dim=1)

Citation

If you use AccPhysBERT, please cite:

@article{Hellert_2025,
  title     = {Domain-specific text embedding model for accelerator physics},
  author    = {Hellert, Thorsten and Montenegro, João and Venturini, Marco and Pollastro, Andrea},
  journal   = {Physical Review Accelerators and Beams},
  volume    = {28},
  number    = {4},
  pages     = {044601},
  year      = {2025},
  publisher = {American Physical Society},
  doi       = {10.1103/PhysRevAccelBeams.28.044601},
  url       = {https://doi.org/10.1103/PhysRevAccelBeams.28.044601}
}

Contact

Thorsten Hellert Lawrence Berkeley National Laboratory 📧 thellert@lbl.gov

Acknowledgments

This model builds on PhysBERT and was trained using NERSC resources. Thanks to Alex Hexemer, Fernando Sannibale, and Antonin Sulc for their support and discussions.