Modelpublic

samodelkini/smollm2-360m-unsloth-essay-scoring-finetune

sourceHugging Faceupdated 6mo agoView on Hugging Face

0likes12downloads

Model Card

smollm2-360m-unsloth-essay-scoring-finetune

LoRA adapter for unsloth/SmolLM2-360M-Instruct trained for the local multi-agent essay grading workflow in this repository.

This is not a standalone full model. It is an adapter intended to be loaded on top of the base model.

Training setup

—Base model: unsloth/SmolLM2-360M-Instruct
—Training method: Unsloth + PEFT LoRA
—Task type: causal LM
—LoRA rank: 8
—LoRA alpha: 16
—LoRA dropout: 0.0
—Target modules: q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj
—Max length: 4096
—Epochs: 2
—Learning rate: 2e-4
—Gradient accumulation: 8
—Quantized loading during training: 4-bit

Training data

The adapter was trained from teacher outputs stored in:

—results/OLMo_PERSUADE_1200_results.json

with essays drawn from:

—corpus/persuade/persuade_sample_1200.json

The run config bundled with this adapter is in distillation_unsloth_run_config.json.

Evaluation

Evaluation was run through the repository's existing multi-agent grading pipeline.

GRE subset

—File: results/original_magic/GRE_essays_smollm2_360m_unsloth_secondversion.json
—n_final: 48
—Final QWK: 0.6092
—Final RMSE: 1.1902
—Exact agreement: 0.3333
—Adjacent agreement: 0.7500

Persuade subset

—File: results/original_magic/persuade_48_smollm2_360m_unsloth_secondversion.json
—n_final: 48
—Final QWK: 0.6720
—Final RMSE: 1.1273
—Exact agreement: 0.3542
—Adjacent agreement: 0.7917

Summary metrics are stored in:

—results/original_magic/smollm2_360m_unsloth_secondversion_numeric_eval_summary.csv

Usage

python

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_model = "unsloth/SmolLM2-360M-Instruct"
adapter_path = "samodelkini/smollm2-360m-unsloth-essay-scoring-finetune"

tokenizer = AutoTokenizer.from_pretrained(adapter_path)
model = AutoModelForCausalLM.from_pretrained(base_model)
model = PeftModel.from_pretrained(model, adapter_path)

Within this repository, the adapter can be evaluated with:

bash

python distillation/evaluate_student.py \
  --input-corpus corpus/GRE/GRE_essays.json corpus/persuade/persuade_48.json \
  --base-model unsloth/SmolLM2-360M-Instruct \
  --adapter-path distillation/checkpoints/smollm2-360m-unsloth-essay-scoring-finetune \
  --output results/student_distilled_unsloth_results.json \
  --model-label smollm2-360m-unsloth-essay-scoring-finetune

Limitations

—This adapter was trained for a specific essay-grading prompt format used in this repo.
—Reported metrics are local evaluation results on the repository's GRE and Persuade subsets, not broad benchmark claims.
—The adapter does not include the full base model weights.