vipsehgal/qwen3-8b-jee-sft

Qwen3-8B JEE SFT

A fine-tuned version of Qwen3-8B specialized for solving IIT JEE Advanced problems in Physics, Chemistry, and Mathematics with detailed chain-of-thought reasoning.

This model was trained via supervised fine-tuning (SFT) using QLoRA on Apple Silicon with MLX.

Model Details

Training Details

• —Framework: MLX on Apple M3 Pro
• —Method: QLoRA — LoRA adapters on a 4-bit quantized base model
• —Data: 3,515 examples (3,163 train / 352 validation)
• —JEEBench CoT: 457 JEE Advanced questions with Claude Opus 4.6-generated step-by-step solutions (Physics, Chemistry, Mathematics)
• —NuminaMath-CoT: 2,706 filtered competition math problems (AMC, AIME, Olympiad-level)

Hyperparameters

Evaluation Results

Evaluated on 200 held-out questions from JEEBench covering Physics, Chemistry, and Mathematics. All models used greedy decoding with max 2,048 tokens.

Key takeaways:

• —SFT improves overall accuracy by +6 percentage points over the base model
• —Largest gain is in Mathematics (+18.2%), driven by competition math training data (NuminaMath-CoT)
• —Chemistry shows a modest improvement (+4.3%)
• —Physics slightly regressed (-4.7%), likely due to the training data being skewed toward mathematics (~83% of examples)

Usage

With MLX on Apple Silicon

pip install mlx-lm

mlx_lm.generate \
    --model vipsehgal/qwen3-8b-jee-sft \
    --prompt "Solve: Find the number of real solutions of x^3 - 3x + 1 = 0"

With Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("vipsehgal/qwen3-8b-jee-sft", torch_dtype="auto", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("vipsehgal/qwen3-8b-jee-sft")

messages = [
    {"role": "system", "content": "You are an expert IIT JEE tutor. Solve problems step-by-step using LaTeX notation. Show all work clearly and arrive at the final answer."},
    {"role": "user", "content": "A particle of mass 2 kg is projected vertically upward with velocity 20 m/s. Find the maximum height reached. (Take g = 10 m/s^2)"}
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=1024)
print(tokenizer.decode(output[0], skip_special_tokens=True))

System Prompt

You are an expert IIT JEE tutor. Solve problems step-by-step using LaTeX notation. Show all work clearly and arrive at the final answer.

Related Models

Limitations

• —Training data is skewed toward mathematics (~83%) vs physics/chemistry (~17%), which limits gains on science subjects
• —Physics performance slightly regressed compared to the base model
• —May produce incorrect reasoning steps while arriving at correct final answers — always verify solutions
• —Evaluated on 200 questions; full 515-question JEEBench benchmark not yet run

License

Apache 2.0 (following the base Qwen3-8B license)