rswaminathan38/llmbench-student-3b-gsm8k-topk-hidden-ft-teacher-20260413

libraryname: transformers pipelinetag: text-generation base_model: meta-llama/Llama-3.2-3B datasets:

• —gsm8k tags:
• —gsm8k
• —transformers
• —vllm
• —text-generation
• —student-model
• —knowledge-distillation ---

Student 3B Top-K KD + Hidden Match

This repo contains the top-k KD with hidden-state matching export for the student model from the GSM8K workflow in this project.

Quick Use

Transformers:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "rswaminathan38/llmbench-student-3b-gsm8k-topk-hidden-ft-teacher-20260413"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto")

vLLM:

vllm serve rswaminathan38/llmbench-student-3b-gsm8k-topk-hidden-ft-teacher-20260413 --dtype auto

Metrics

• —Test relaxed exact-match accuracy: 0.2252
• —Correct / examples: 297 / 1319
• —Avg generated tokens: 253.0349
• —Prompt style used during evaluation: cot_step_by_step

Training Details

• —Base model: meta-llama/Llama-3.2-3B
• —Variant: top-k KD with hidden-state matching
• —Output source: /storage/ice1/3/3/rswaminathan38/LLM_Bench/LLMOptimization/Model_Optimizations/outputs/hf_topk_hidden_teacher_2026-04-13
• —Teacher used for distillation: Fine-tuned GSM8K teacher 8B (2026-04-10)
• —num_train_epochs: 10
• —per_device_train_batch_size: 2
• —gradient_accumulation_steps: 8
• —learning_rate: 1e-05
• —warmup_ratio: 0.05
• —max_seq_length: 1024
• —alpha: 0.7
• —temperature: 2.0
• —kd_method: topk
• —top_k: 64
• —hidden_match: True
• —hidden_match_weight: 0.05

Notes

• —Prompt format in this repo is question + "\n\nLet's think step by step.\n".
• —Original Meta Llama license and access requirements still apply to downstream use.
• —run_config.json and eval/test_summary.json are uploaded alongside the weights when available.