debashis2007/healthcare-gpt2

Healthcare GPT-2 LoRA Model

A fine-tuned GPT-2 model specialized for healthcare and medical text generation using LoRA (Low-Rank Adaptation).

Model Details

• —Model Name: healthcare-gpt2-lora
• —Base Model: gpt2
• —Model Type: LoRA Adapter (PEFT)
• —Fine-tuning Method: LoRA (Low-Rank Adaptation)
• —Language: English
• —Domain: Healthcare, Medical

Model Description

This model is a fine-tuned version of GPT-2, optimized for healthcare-related text generation. It has been trained on healthcare datasets including clinical notes, medical Q&A, medication information, and other medical text data.

The model uses LoRA (Low-Rank Adaptation) for efficient fine-tuning, allowing it to be trained with minimal computational resources while maintaining the base model's general language capabilities.

Uses

Direct Use

This model is intended for:

• —Generating healthcare-related text
• —Medical documentation assistance
• —Clinical note generation
• —Medical Q&A systems
• —Healthcare chatbot applications

Example Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# Load base model
base_model = AutoModelForCausalLM.from_pretrained("gpt2")
tokenizer = AutoTokenizer.from_pretrained("gpt2")

# Load LoRA adapter
model = PeftModel.from_pretrained(base_model, "./cpu_healthcare_output")

# Generate text
prompt = "Patient presents with:"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_length=200, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Training Details

Training Data

The model was fine-tuned on healthcare datasets including:

• —Clinical notes
• —Medical Q&A pairs
• —Medication information
• —Lab results and vital signs
• —Medical history records

Training Hyperparameters

• —LoRA Rank (r): 8
• —LoRA Alpha: 16
• —LoRA Dropout: 0.1
• —Learning Rate: 5e-5
• —Batch Size: 1
• —Max Length: 128-256 tokens

Limitations

• —This is a small model (GPT-2 base) optimized for CPU training
• —For better performance, consider using larger models like Mistral-7B with GPU
• —The model should not be used for actual medical diagnosis or treatment decisions
• —Always verify medical information with qualified healthcare professionals

Recommendations

• —Use this model as an assistant tool, not as a replacement for medical professionals
• —Verify all generated medical information
• —Consider fine-tuning on domain-specific datasets for specialized use cases
• —For production use, consider using larger models trained on more extensive datasets

Citation

If you use this model, please cite:

@misc{healthcare-gpt2-lora,
  title={Healthcare GPT-2 LoRA Model},
  author={Fine-tuned for healthcare applications},
  year={2024},
  base_model={gpt2}
}

Framework Versions

• —PEFT 0.17.1
• —Transformers 4.57.1
• —PyTorch 2.9.0