CoolFace
Modelpublic

Proteinrequired/email-triage-lora

sourceHugging Facellama3.2updated 3mo agoView on Hugging Face
0likes32downloads
Model Card

Email-Triage LoRA (Llama-3.2-3B-Instruct)

A LoRA adapter (r=16) fine-tuned via Behavioral Cloning to triage corporate emails by selecting one of three tools — route_to_human, auto_reply, or ask_for_clarification — returned as a strict JSON tool call.

  • —Base model: unsloth/Llama-3.2-3B-Instruct
  • —Method: Behavioral Cloning (Unsloth + Hugging Face TRL), 60 steps
  • —Adapter: LoRA, r=16
  • —Project: Enterprise Email Triage Simulator
  • —Live demo (Gemini-backed UI): https://huggingface.co/spaces/Proteinrequired/enterprise-email-triage-v2

Results — held-out tool-selection accuracy

Evaluated on a 21-email held-out test set (canonical 100-email dataset, intent-stratified 80/20 split, seed 42; BC trained only on the 79 train emails). Metric = did the policy pick the reward system's optimal tool for each email.

PolicyHeld-out accuracy (N=21)
Random choice33.0%
Always route_to_human47.6%
Rule-based heuristic76.2% (16/21)
This adapter71.4% (15/21)

The adapter is perfect on every `route_to_human` intent but over-escalates routine (auto_reply) and ambiguous (ask_for_clarification) mail — a known Behavioral-Cloning artifact (trained on reward-positive rollouts skewed toward escalation). It lands within one example of a hand-tuned rule baseline and far above the random / always-escalate floors. Reproduce with `run_trained_eval.py`.

Usage

python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = "unsloth/Llama-3.2-3B-Instruct"
adapter = "Proteinrequired/email-triage-lora"

tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, device_map="auto")
model = PeftModel.from_pretrained(model, adapter)

The model expects the system/user prompt format defined in `run_trained_eval.py` and responds with a JSON object: {"tool": "<route_to_human|auto_reply|ask_for_clarification>"}.

Limitations

  • —Trained on synthetic corporate emails; not validated on real inboxes or other domains.
  • —3B model — needs a GPU for low-latency inference. The live demo uses Gemini via API for CPU-only hosting; this adapter is the project's open-weights research artifact.
  • —Optimizes the project's reward policy; not a general-purpose safety/abuse classifier.