NeuML/astrobert-small

AstroBERT Small

This is a 22.7M parameter BERT encoder-only model trained on ArXiv abstracts categorized as astro-ph and Wikipedia articles labeled as astronomy related.

This is a domain-specialized small model that often performs as good as models 10-100x larger. It demonstrates that narrowing down a model to a small domain requires less overall parameters than models generalized for all problems.

Usage

astrobert-small can be loaded using Hugging Face Transformers as follows.

from transformers import AutoModel

model = AutoModel.from_pretrained("neuml/astrobert-small")

The model is intended to be further fine-tuned for a specific task such as Text Classification, Entity Extraction, Sentence Embeddings and so on.

More Information

Read more about the model in this article.