CoolFace
Modelpublic

kleinpanic93/canvas-calendar-agent-v7-dpo

sourceHugging Faceapache-2.0updated 5mo agoView on Hugging Face
0likes38downloads
Model Card

Canvas Calendar Agent v7 — DPO

Fine-tuned google/gemma-4-E2B-it (2.7B params) for the Canvas Calendar Agent project. Trained with Direct Preference Optimization (Rafailov et al., 2023, arXiv:2305.18290) on top of an SFT-trained checkpoint.

Intended use

Reads a student's Canvas LMS state (assignments, courses, deadlines) and their calendar, then produces concrete, actionable scheduling plans (study blocks, exam prep, rescheduling). Speaks the native Gemma-4 tool-call format (<|tool_call>call:tool.name{args}<tool_call|>) for 18 Canvas/Calendar/Study tools.

Training procedure

StageMethodDataWall time
SFTFull-parameter fine-tuning181 trajectory rows from Canvas sessions~12 min
DPOβ=0.1, sigmoid loss, frozen-snapshot reference1,071 preference pairs labeled by Gemma-4-31B-IT-NVFP4 teacher (3-vote majority at temp=0)9:03

DPO hyperparameters

python
DPOConfig(
    loss_type="sigmoid",          # original DPO loss (Rafailov §4)
    beta=0.1,                     # KL implicit coefficient
    num_train_epochs=1,
    per_device_train_batch_size=1,
    gradient_accumulation_steps=8,  # effective batch = 8
    learning_rate=5e-6,
    bf16=True,
    sync_ref_model=False,         # freeze ref at init
    precompute_ref_log_probs=True,
    optim="adamw_torch",
)

Reference policy: snapshot of policy (= SFT model) at training init, frozen via precompute_ref_log_probs=True. Mathematically equivalent to the paper's π_ref = π_SFT.

Training metrics

MetricFinal value
train_loss0.2229
rewards/accuracies0.9032
rewards/margins5.142
rewards/chosen-3.362
rewards/rejected-8.504
logps/chosen-238.5
logps/rejected-328.9
entropy1.01

Random preference accuracy is 0.5; our DPO model correctly ranks 90.3% of held-out preference pairs.

Model architecture

  • —Base: google/gemma-4-E2B-it (2.7B params, decoder-only Transformer)
  • —Precision: bf16 weights, full-parameter fine-tune (no LoRA / no quantization)
  • —Context: 4,096 tokens during training, 8,192 supported at inference
  • —Tool-call format: native Gemma-4 delimiters (<|tool_call>call:NAME{ARGS}<tool_call|>)

Hardware

NVIDIA DGX Spark (Grace-Blackwell GB10 SoC, 122 GiB unified memory, SM121, aarch64). Single GPU. Container: nvcr.io/nvidia/pytorch:25.11-py3. TRL 1.1, Transformers 4.47+, PyTorch 2.10 nightly.

Usage

python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "kleinpanic93/canvas-calendar-agent-v7-dpo",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("kleinpanic93/canvas-calendar-agent-v7-dpo")

messages = [
    {"role": "system", "content": "You are a Canvas calendar agent. ..."},
    {"role": "user", "content": "What assignments do I have due this week?"},
]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True).to(model.device)
out = model.generate(inputs, max_new_tokens=512, do_sample=False)
print(tokenizer.decode(out[0][inputs.shape[1]:], skip_special_tokens=False))

The output will contain <|tool_call>...<tool_call|> delimiters that the `canvas_sdk.tool_parser` decodes into structured tool calls.

For end-to-end agent usage with auto-download:

bash
pip install canvas-sdk[autodownload]
python -m canvas_sdk.demo "Plan my study schedule for next week"

Data ethics + PII handling

Training data was anonymized through a two-pass process before inclusion:

  1. 1.CRN scrubbing: course identifiers replaced with stable @COURSE_n tokens.
  2. 2.PII pass: emails, phones, professor/student names, building/room references replaced with @PROF_EMAIL/@CONTACT_EMAIL/@PHONE/@PROFn/@STUDENTn/@LOCn. spaCy NER + custom regex.

The training data is the companion dataset.

Limitations

  • —Only trained on 1,071 preference pairs and 181 trajectory rows. Coverage of diverse Canvas course structures is limited.
  • —All training contributors used Virginia Tech instances of Canvas LMS — generalization to other institutions is untested.
  • —Tool-call format is Gemma-4 specific. Will not work with chat templates that don't preserve <|tool_call>...<tool_call|> delimiters.
  • —2.7B model — qualitative reasoning ability is below larger models. The 31B-IT teacher (used for labeling) is sometimes too good at distinguishing pairs the smaller policy cannot perfectly reproduce.

Citation

If you use this model, please cite:

bibtex
@article{rafailov2023dpo,
  title={Direct Preference Optimization: Your Language Model is Secretly a Reward Model},
  author={Rafailov, Rafael and Sharma, Archit and Mitchell, Eric and Ermon, Stefano and Manning, Christopher D and Finn, Chelsea},
  journal={NeurIPS},
  year={2023},
  url={https://arxiv.org/abs/2305.18290}
}

License

Apache 2.0 (matches Gemma-4 base license).

Project