mkdiscovery/Pharma-DrugInteraction-Qwen-0.5B

💊 Pharma-DrugInteraction-Qwen-0.5B

Pharmaceutical Drug-Interaction Learning Prototype

A small domain-specific language model prototype created by fine-tuning Qwen/Qwen2.5-0.5B-Instruct with LoRA (Low-Rank Adaptation) on a curated pharmaceutical drug-interaction dataset.

Version: v1.0 Base Model: Qwen/Qwen2.5-0.5B-Instruct Fine-Tuning: LoRA Model Size: 0.5B parameters Purpose: Educational / Research Status: Learning Prototype

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
import torch

BASE_MODEL = "Qwen/Qwen2.5-0.5B-Instruct"
ADAPTER = "mkdiscovery/Pharma-DrugInteraction-Qwen-0.5B"

device = "cuda" if torch.cuda.is_available() else "cpu"

print("Loading model...")

tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)

model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL,
    torch_dtype=torch.float16 if device == "cuda" else torch.float32,
    device_map="auto" if device == "cuda" else None,
)

model = PeftModel.from_pretrained(model, ADAPTER)
model.eval()

print("Model loaded!")
print("Ask questions below. Type 'exit' to quit.\n")


while True:

    question = input("You: ")

    if question.lower() in ["exit", "quit", "q"]:
        print("Bye!")
        break

    messages = [
        {
            "role": "user",
            "content": question
        }
    ]

    text = tokenizer.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=True,
    )

    inputs = tokenizer(
        text,
        return_tensors="pt"
    ).to(model.device)

    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            max_new_tokens=256,
            temperature=0.2,
            do_sample=True,
        )

    response = tokenizer.decode(
        outputs[0][inputs["input_ids"].shape[1]:],
        skip_special_tokens=True
    )

    print(f"Model: {response}\n")

🚀 What is this?

This project explores how a small Large Language Model (LLM) can be adapted toward the pharmaceutical drug-interaction domain using parameter-efficient fine-tuning.

Instead of training an LLM from scratch, this project takes an existing instruction-tuned model:

Qwen/Qwen2.5-0.5B-Instruct

and trains a small set of additional LoRA parameters using pharmaceutical drug-interaction examples.

Architecture

                 Qwen 0.5B
                    │
                    │
                    ▼
          ┌──────────────────┐
          │   LoRA Adapter   │
          │   Fine-Tuning    │
          └────────┬─────────┘
                   │
                   ▼
        Pharma-DrugInteraction
               Qwen-0.5B
                   │
                   ▼
          Drug Interaction
             Questions
                   │
                   ▼
              Response

🎯 Project Goal

The goal of V1 was not to build a clinical-grade medical system.

The goal was to practically understand:

• —How LLMs work
• —How datasets are prepared for fine-tuning
• —How pharmaceutical data can be normalized
• —How different datasets can be combined
• —How conversational training data is generated
• —How LoRA fine-tuning works
• —How a fine-tuned model behaves compared with the original model
• —What limitations appear when using a very small model

This is primarily a hands-on AI/ML learning project.

🧠 Model

Base Model

Qwen/Qwen2.5-0.5B-Instruct

The base model provides the general language understanding and generation capabilities.

Fine-Tuning Method

LoRA — Low-Rank Adaptation

Instead of updating all parameters of the Qwen model, LoRA adds trainable low-rank matrices to selected layers.

Conceptually:

Original Qwen Model
       │
       ├── Frozen Parameters
       │
       └── LoRA Parameters
                 │
                 ▼
        Pharmaceutical Domain

This makes the training process significantly lighter than full-model fine-tuning.

📊 Dataset

The training data was created from pharmaceutical/drug-interaction datasets.

The preparation pipeline included:

Raw Pharmaceutical Data
          │
          ▼
Drug Name Normalization
          │
          ▼
Drug Matching
          │
          ▼
Interaction Records
          │
          ▼
Conversational Training Examples
          │
          ▼
train.jsonl

For this V1 prototype, the pharmaceutical drug selection was intentionally limited to a smaller subset because of available compute resources.

V1 scope

• —Top 100 drugs
• —13,534 matched interaction records
• —Multiple conversational question variations were generated for the interaction records
• —Final training file: train.jsonl

The smaller scope was intentional so the project could be trained and tested on available hardware.

🧪 Training Format

The training examples use a conversational format compatible with instruction-tuned models.

Example:

{
  "messages": [
    {
      "role": "user",
      "content": "Does Zolpidem Tartrate interact with Itraconazole?"
    },
    {
      "role": "assistant",
      "content": "Interaction: increase drug exposure.\n\nItraconazole increased the exposure of zolpidem..."
    }
  ]
}

Multiple question formulations were generated for interaction records to expose the model to different ways a user might ask about the same drug interaction.

⚙️ Training Configuration

🔬 Before vs After Fine-Tuning

One of the main objectives of this project was to compare the base model with the fine-tuned model.

Before Fine-Tuning

The original Qwen model could generate general pharmaceutical-looking responses, but it could also:

• —misunderstand drug names
• —confuse medications
• —generate unsupported explanations
• —produce generic responses

After Fine-Tuning

The model became more aligned with the structure and terminology of the drug-interaction training data.

It learned patterns such as:

User:
Does Drug A interact with Drug B?

Model:
Interaction: <interaction category>

<supporting pharmaceutical text>

However, the V1 model can still produce incorrect or mismatched evidence.

This is an important limitation of using fine-tuning alone for precise pharmaceutical knowledge retrieval.

⚠️ Limitations

This model is a learning/research prototype.

It should not be considered a reliable medical information system.

Known limitations include:

• —Small model size: 0.5B parameters
• —Limited V1 drug coverage
• —Limited training compute
• —Possible hallucinations
• —Possible incorrect drug-pair associations
• —Fine-tuning does not guarantee exact factual retrieval
• —Training data may contain inconsistencies
• —No retrieval/database verification layer
• —No clinical validation
• —No medical professional verification of generated responses

The model should therefore not be used for diagnosis, prescribing, dosage decisions, or clinical decision-making.

Always verify drug-interaction information using authoritative pharmaceutical references and qualified healthcare professionals.

💻 Quick Start

1. Install dependencies

pip install torch transformers peft

Depending on the environment, compatible versions of the Hugging Face ecosystem may also be required.

2. Download the base model

The model is based on:

Qwen/Qwen2.5-0.5B-Instruct

The LoRA repository contains the adapter rather than a complete copy of the base model.

3. Load the LoRA adapter

The included inference.py loads:

Qwen/Qwen2.5-0.5B-Instruct
             +
     LoRA Adapter
             ↓
Pharma-DrugInteraction-Qwen-0.5B

Run:

python3 inference.py

🧑‍💻 Example

💊 You: Does Zolpidem Tartrate interact with Itraconazole?

🤖 Assistant:

Interaction: increase drug exposure.

Itraconazole increased the exposure of zolpidem...

Type:

exit

to close the application.

📁 Repository Structure

Pharma-DrugInteraction-Qwen-0.5B/
│
├── adapter_config.json
├── adapter_model.safetensors
│
├── tokenizer_config.json
├── tokenizer.json
├── special_tokens_map.json
├── added_tokens.json
├── merges.txt
├── vocab.json
│
├── inference.py
├── README.md
└── LICENSE

Important

adapter_model.safetensors contains the trained LoRA adapter weights.

The complete Qwen base model is not duplicated in this repository.

🔧 How the Model Works

At inference time:

User Question
      │
      ▼
Qwen Tokenizer
      │
      ▼
Qwen 0.5B Base Model
      │
      +
      │
LoRA Adapter
      │
      ▼
Generated Response

The LoRA adapter modifies the behavior of the base model toward patterns learned from the pharmaceutical training examples.

🧪 Why LoRA?

Full fine-tuning would require updating the entire model.

LoRA instead trains a relatively small number of additional parameters while keeping the original model largely frozen.

This makes it particularly useful for:

• —learning experiments
• —smaller compute environments
• —domain adaptation
• —rapid prototyping
• —parameter-efficient fine-tuning

🛣️ Future Work

Possible future versions could explore:

V2 — Larger Dataset

Increase drug coverage beyond the V1 top-100 selection.

V3 — Retrieval-Augmented Generation

Introduce a retrieval layer so that the model can retrieve the exact drug-interaction record instead of relying entirely on information encoded during fine-tuning.

Question
   ↓
Drug Pair Retrieval
   ↓
Relevant Evidence
   ↓
LLM
   ↓
Answer

V4 — Evaluation

Build a dedicated evaluation dataset and measure:

• —Exact-match accuracy
• —Interaction-category accuracy
• —Drug-pair coverage
• —Hallucination rate
• —Seen vs unseen pair performance

📌 Model Card Summary

🙏 Acknowledgements

This project was created as a hands-on exploration of LLMs, pharmaceutical datasets, data preprocessing, and parameter-efficient fine-tuning.

The project builds upon the capabilities of:

• —Qwen
• —Hugging Face Transformers
• —Hugging Face Datasets
• —PEFT / LoRA

⚖️ Disclaimer

Educational and research purposes only.

This model is not a medical device and has not been clinically validated.

Generated responses may be inaccurate, incomplete, or misleading. Do not use this model as a substitute for professional medical advice, prescribing information, official drug labels, or validated drug-interaction databases.

For real-world medical decisions, consult qualified healthcare professionals and authoritative pharmaceutical references.

👥 Contributors

This project was a collaborative effort combining pharmaceutical research and AI engineering.

Ayushi Nair

Concept, Research, Data Collection & Validation

• —Led the project concept and research direction
• —Collected and curated pharmaceutical datasets
• —Validated data quality and interaction records

LinkedIn: <https://www.linkedin.com/in/ayushi--nair/>

Midhun Krishna

Engineering & Infrastructure

• —Dataset preprocessing and preparation
• —LoRA fine-tuning pipeline
• —Model training and inference
• —Hugging Face model packaging and deployment
• —Repository development and documentation

LinkedIn: <https://www.linkedin.com/in/midhunvellarakkad/>