CoolFace
Modelpublic

Hodfa71/olmo-1b-lume-pstu

sourceHugging Faceapache-2.0updated 6mo agoView on Hugging Face
0likes13downloads
Model Card

olmo-1b-lume-pstu

OLMo-1B after PSTU unlearning on the LUME benchmark. Removes all memorized PII (0% QA accuracy) with minimal PPL impact (+0.9%).

Model Details

This model is the result of applying PSTU (Per-Secret-Type Unlearning) to an OLMo model infected with synthetic PII from the LUME benchmark.

LUME Benchmark

LUME (Language Model Unlearning Made Easy) provides OLMo models fine-tuned on 250 synthetic biographies containing PII (DOB, SSN, phone, email, address).

Evaluation metrics:

  • —QA Accuracy: Fraction of PII recoverable via QA prompts (lower is better)
  • —ROUGE-L: Overlap with memorized biographies
  • —PPL: WikiText-2 perplexity

Results

MethodQA ↓R-L ↓PPL ↓ΔPPL
Infected100%1.0varies---
PSTU0%~0.1~clean<2%

Usage

python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Hodfa71/olmo-1b-lume-pstu")
tokenizer = AutoTokenizer.from_pretrained("Hodfa71/olmo-1b-lume-pstu")

Related Models

Citation

If you use this model, please cite our work on Per-Secret-Type Unlearning (PSTU).