OscarYanez85/biomed-gpt-pubmed-small

Biomed-GPT-PubMed-Small

Overview

Biomed-GPT-PubMed-Small is a lightweight GPT-style transformer trained from scratch on a sampled PubMed abstract corpus.

This model was built as a local biomedical language modeling project using:

• —a custom SentencePiece tokenizer
• —a GPT-style transformer architecture implemented in PyTorch
• —a PubMed abstract corpus pipeline
• —local GPU training on consumer hardware
• —Hugging Face Hub integration via PyTorchModelHubMixin

The project is intended for educational, experimental, and portfolio purposes in biomedical NLP and LLM engineering.

Architecture

• —Model type: decoder-only GPT-style transformer
• —Context length: 128
• —Embedding dimension: 256
• —Number of layers: 6
• —Number of attention heads: 8
• —Vocabulary size: 8000
• —Framework: PyTorch

Training Data

The model was trained on a sampled and cleaned subset of PubMed abstracts.

Training text was constructed from biomedical abstract content and processed into a local corpus for custom tokenizer training and autoregressive next-token prediction.

Training Setup

• —Tokenizer: SentencePiece BPE
• —Vocabulary size: 8000
• —Training objective: next-token prediction
• —Mixed precision: enabled
• —Validation tracking: enabled
• —Gradient accumulation: enabled
• —Hardware: NVIDIA GeForce RTX 3070 Laptop GPU (8GB VRAM)

Evaluation

Final observed validation metrics during training:

• —Validation loss: 2.7568
• —Perplexity: 15.75

These metrics should be interpreted as internal language-modeling metrics only, not as clinical performance indicators.

Intended Use

This model is intended for:

• —biomedical NLP experimentation
• —local language model prototyping
• —educational transformer training workflows
• —Hugging Face model packaging demonstrations
• —portfolio and research engineering work

Limitations

• —This is a small model trained on a sampled subset of PubMed abstracts.
• —It is not instruction-tuned.
• —It may hallucinate, drift semantically, or produce incorrect biomedical statements.
• —It is not grounded to citations or retrieval.
• —It is not a medical question-answering system.
• —It is not a clinical reasoning system.

The model is better suited for biomedical text continuation than direct question answering.

Medical Disclaimer

This model is not validated for clinical use.

It must not be used for:

• —diagnosis
• —treatment recommendation
• —triage
• —patient counseling
• —clinical decision-making
• —emergency or safety-critical workflows

It is a research and educational model only.

Example Prompts

Better prompt styles for this model include:

• —Pneumonia is
• —Hypertension is associated with
• —The objective of this study was
• —In patients with asthma,

The base model is optimized for biomedical-style continuation, not instruction following.

Future Work

Planned next steps include:

• —instruction tuning on biomedical QA pairs
• —improved local chat-style prompting
• —larger tokenizer vocabulary
• —larger PubMed training corpora
• —retrieval augmentation
• —benchmark evaluation on biomedical QA tasks

Author

Oscar Yanez

Biomedical AI / Python / Local LLM Engineering