NeuML/biomedbert-small

BiomedBERT Small

This is a 22.7M parameter BERT encoder-only model trained on data from PubMed. The raw data was transformed using PaperETL with the results stored as a local dataset via the Hugging Face Datasets library.

This model is designed to be a solid-performing small model fitting in between the 110M parameter BiomedBERT Base model and the tiny BiomedBERT Hash series of models.

Usage

biomedbert-small can be loaded using Hugging Face Transformers as follows.

from transformers import AutoModel

model = AutoModel.from_pretrained("neuml/biomedbert-small")

The model is intended to be further fine-tuned for a specific task such as Text Classification, Entity Extraction, Sentence Embeddings and so on.

Evaluation Results

This Medical Abstracts Text Classification Dataset was used to evaluate the model's performance. A handful of biomedical models and general models were selected for comparison.

Metrics were generated using Hugging Face's standard run_glue script as shown below.

python run_glue.py --model_name_or_path neuml/biomedbert-small --dataset-name medclassify --do_train --do_eval --max_seq_length 128 --per_device_train_batch_size 32 --learning_rate 1e-4 --num_train_epochs 4 --output_dir outputs --trust-remote-code True

Note: The original dataset was saved locally as `medclassify` the the `conditionlabel column renamed to label` to work more easily with the glue script_

As we can see, this model performs very well against models much larger in size. This dataset is a challenging one!

More Information

Read more about the model in this article.