piyushgupta53/smollm2-135m-english-to-latex-notation-lora

SmolLM2 135M English-to-LaTeX Notation — rsLoRA Adapter

A rank-32 rsLoRA adapter for HuggingFaceTB/SmolLM2-135M-Instruct that converts a precise English description of mathematical notation into one standalone LaTeX expression.

This model writes notation; it is not intended to solve, calculate, prove, derive, or explain mathematics.

Intended output contract

• —Input: one precise English request for mathematical notation.
• —Output: exactly one standalone LaTeX expression.
• —No dollar signs, code fences, prose, answer labels, or document wrappers.

Usage

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

adapter_id = "piyushgupta53/smollm2-135m-english-to-latex-notation-lora"
base_id = "HuggingFaceTB/SmolLM2-135M-Instruct"

system_prompt = (
    "Convert the user's English description of mathematical notation into LaTeX. "
    "Return exactly one standalone LaTeX expression. Do not include dollar signs, "
    "code fences, explanations, or any other text."
)

tokenizer = AutoTokenizer.from_pretrained(adapter_id)
base_model = AutoModelForCausalLM.from_pretrained(
    base_id,
    torch_dtype="auto",
    device_map="auto",
)
model = PeftModel.from_pretrained(base_model, adapter_id)

messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": "Write the second derivative with respect to g of the fourth power of q evaluated at g."},
]
inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt",
).to(model.device)

with torch.inference_mode():
    generated = model.generate(
        inputs,
        max_new_tokens=96,
        do_sample=False,
        pad_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(generated[0, inputs.shape[-1]:], skip_special_tokens=True).strip())

Expected form:

\frac{d^2}{dg^2}q\left(g\right)^{4}

Training

• —Base model: HuggingFaceTB/SmolLM2-135M-Instruct
• —Start point: clean base model, not a previous adapter
• —Method: response-only supervised fine-tuning with rank-32 rsLoRA
• —Train / validation rows: 22,460 / 2,496
• —Formal targets were grouped before splitting to prevent target leakage
• —Epochs: 3; selected checkpoint: epoch 3
• —Learning rate: 2e-4
• —Weight decay: 0.01
• —Warmup: 5%
• —Precision: bfloat16
• —Seed: 20260818
• —LoRA alpha: 32; dropout: 0.05
• —Target modules: attention projections and MLP projections
• —Adapter weights SHA-256: bb8086d19ef704e4d195ea4125ea800edb1a9623a7fd49b3472904c500cba60a

The 24,956-row training corpus combines deterministic formal targets with audited English paraphrases.

Companion artifacts

• —Dataset: piyushgupta53/english-to-latex-notation-sft
• —Standalone merged model: piyushgupta53/smollm2-135m-english-to-latex-notation

Evaluation

The checkpoint was evaluated greedily with the system prompt above and max_new_tokens=96. Both evaluation sets were frozen and excluded from training.

Diagnostic semantic pass by arm:

• —Wording shift: 73/100
• —Novel structural coverage: 61/100

Non-exact predictions were screened with DeepSeek V4 Flash and then every non-exact row was adjudicated with DeepSeek V4 Pro. Compilation and semantic correctness were measured separately: compilable LaTeX can still express the wrong meaning.

Limitations

This is a narrow 135M-parameter model and is not reliable enough for unreviewed high-stakes use. It still fails many hard or unfamiliar structures. In the structural diagnostic it scored 0/10 on augmented matrices and 0/10 on indexed piecewise expressions with an otherwise branch. Always validate syntax and meaning when correctness matters.

The model may also produce a valid expression that is equivalent to, but textually different from, a reference. Exact match alone is therefore insufficient for comprehensive evaluation.

License

Apache-2.0, consistent with the base model. See the base model card for its terms and limitations.