imuki04/Qwen2.5-0.5B-OpenBookQA-Finetuned

Qwen2.5-0.5B - OpenBookQA Full Fine-Tuned

This is a full fine-tuned version of the Qwen/Qwen2.5-0.5B causal language model on the allenai/openbookqa dataset. It has been specifically trained to perform multiple-choice question answering for elementary-level science facts and reasoning.

Model Description

• —Model Type: Causal Language Model
• —Language: English
• —Base Model: Qwen/Qwen2.5-0.5B
• —Training Paradigm: Full Fine-Tuning (all 0.5 billion parameters updated)
• —Task: Multiple-Choice Question Answering (MCQA)

Dataset and Prompt Format

The model was fine-tuned entirely on the OpenBookQA dataset.

Inputs must be formatted exactly as they were during training to achieve optimal results. The prompt ends with Answer: and the model predicts a single token (A, B, C, or D) corresponding to the correct choice.

Prompt Template:

Question: {question_stem}
A. {choice_1}
B. {choice_2}
C. {choice_3}
D. {choice_4}
Answer: 

Example Usage:

Question: The sun is responsible for
A. puppies learning new tricks
B. children growing up and getting old
C. flowers wilting in a vase
D. plants sprouting, blooming and wilting
Answer: 

Training Hyperparameters

The following hyperparameters were utilized during the full fine-tuning run:

• —Optimizer: AdamW
• —Precision: bfloat16
• —Epochs: 3
• —Learning Rate: 1e-5
• —Scheduler: Cosine configuration
• —Warmup Ratio: 0.1 (10% of total steps)
• —Weight Decay: 0.01
• —Batch Size: 16 (Per-Device: 4, Gradient Accumulation: 4)
• —Gradient Checkpointing: Enabled (for memory stability)

Limitations and Bias

Since this is a primarily 0.5-billion parameter model fine-tuned on a narrow academic science dataset, it functions best as an exploratory project rather than a generalized reasoner. It might hallucinate off-topic responses if queried outside the strictly formatted MCQA template.

Usage

You can load the model directly using transformers:

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "imuki04/Qwen2.5-0.5B-OpenBookQA-Finetuned" # Replace your repo name if needed

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

prompt = "Question: The sun is responsible for\nA. puppies learning new tricks\nB. children growing up and getting old\nC. flowers wilting in a vase\nD. plants sprouting, blooming and wilting\nAnswer:"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=5, temperature=0.0)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))