CoolFace
Datasetpublic

EdyVision/pii-skills-ablation-results

PII Skills Ablation — Scored Results This repository contains model predictions and evaluation scores for the ablation study described in: "Asymmetry, Not Capability: Evaluation Shapes Tool-Augmented PII Detection in Small Language Models" Results are produced by running four open-weight instruction-tuned models (Gemma 2 9B, Llama 3.1 8B, Mistral 7B, Qwen 2.5 7B) under four primary conditions (zero-shot, +Docs, +Tool, +Skills), plus three baselines (standalone PII-Codex detector… See the full description on the dataset page: https://huggingface.co/datasets/EdyVision/pii-skills-ablation-results.

sourceHugging Facemitupdated 2mo agoView on Hugging Face
0likes60downloads
Dataset Card

PII Skills Ablation — Scored Results

This repository contains model predictions and evaluation scores for the ablation study described in:

"Asymmetry, Not Capability: Evaluation Shapes Tool-Augmented PII Detection in Small Language Models"

Results are produced by running four open-weight instruction-tuned models (Gemma 2 9B, Llama 3.1 8B, Mistral 7B, Qwen 2.5 7B) under four primary conditions (zero-shot, +Docs, +Tool, +Skills), plus three baselines (standalone PII-Codex detector, few-shot, chain-of-thought) and two robustness controls (a 14B model and a full-precision re-run), on the benchmark in EdyVision/pii-skills-ablation, with scoring against PII-Codex-aligned ground truth.

Dataset Summary

Overview

This dataset holds model predictions and per-sample evaluation scores (e.g. F1, precision, recall) for the ablation study. One row corresponds to one (sample, model, condition) run. Schema is fixed across splits for consistent loading.

Key Characteristics

  • —One row per (sample, model, condition)
  • —Predictions stored as JSON lists of spans; scores as JSON strings
  • —Splits by run type and prompt version (e.g. main_v1, baselines_v1, scaling_v1)

Dataset Description

  • —Purpose: Hosting of predictions and per-sample/per-condition scores for reproducibility and secondary analysis of the ablation study.
  • —Contents: One row per (sample, model, condition) with model outputs, scores (e.g. F1), and metadata (tool use, skill view, timing).
  • —Splits: Splits follow run type and prompt version. Schema is fixed across splits for consistent loading.

Dataset Structure

ColumnTypeDescription
sample_idstringIdentifier for the benchmark sample (matches benchmark id).
sourcestringSource dataset: ai4privacy, nvidia_nemotron_pii, or gretel_pii_masking.
modelstringModel identifier: gemma2_9b, llama3_8b, mistral_7b, qwen2_7b, qwen2_14b (scaling control), or pii_codex (standalone detector).
conditionstringCondition: zero_shot, with_docs, with_tools, with_skills, few_shot, cot, or detector.
prompt_versionstringPrompt version used (e.g. v1).
run_typestringRun type: pilot, main, detector, baselines, fp16, or scaling.
predictionsstringJSON list of model-predicted spans (see span structure below).
scoresstringJSON string of evaluation metrics (e.g. F1, precision, recall) for that sample.
errorstringError message if the run failed; empty string otherwise.
tool_executedboolWhether a tool was invoked (for +Tool / +Skills).
skill_viewedboolWhether the Skill document was used (for +Skills).
elapsed_secondsfloatWall-clock time for the run.
conversation_turnsintNumber of turns (for multi-turn conditions).

Scores are computed against the benchmark ground truth after a single symmetric label alignment to PII-Codex types, applied identically to predictions and ground truth (see paper and code repo).

Prediction span structure

The predictions column is a JSON-encoded list of spans. Each span has the structure:

json
{
  "type": "EMAIL_ADDRESS",
  "text": "john.smith@acme.com",
  "start": 22,
  "end": 41
}

type is a PII-Codex type name, text is the exact predicted span, and start/end are character offsets into the sample text. Parse the column with json.loads() before use.

Splits

SplitRowsWhat it is
main_v132,000Primary ablation: 4 models × 4 conditions × 2,000 samples (4-bit).
pilot_v13,200Pilot ablation: 4 models × 4 conditions × 200 samples (4-bit).
baselines_v116,000Few-shot and chain-of-thought baselines: 4 models × 2 conditions × 2,000 samples.
detector_v12,000Standalone PII-Codex/Presidio detector (no model) on the 2,000 samples.
fp16_v14,800Full-precision (fp16/bf16) control: 4 models × 4 conditions × 300-sample subset.
scaling_v18,000Scale control: Qwen2.5-14B (4-bit) × 4 conditions × 2,000 samples.

Intended Use

  • —Primary: Reproducibility and extended analysis for the ablation study (aggregate statistics, plots, and tables in the paper are produced from these results).
  • —Secondary: Re-analysis, meta-analysis, or comparison with other PII detection systems on the same benchmark. Not intended as training data.
  • —Out-of-Scope Uses: Not intended as training data. Use only with the same benchmark and alignment for comparable numbers.

Citations

If you use these results, please cite the paper and, where applicable, the results dataset:

bibtex
@article{rosado2026context,
  title   = {PII Agent Skills Ablation Model Outputs and Scores},
  author  = {Rosado, Eidan J.},
  year    = {2026},
  note    = {Scored results: \url{https://huggingface.co/datasets/EdyVision/pii-skills-ablation-results}}
}

Related citations

If you also use the benchmark or run experiments on it, please cite the benchmark and its source datasets as described in the benchmark datacard.

Ethical Considerations

  • —Results contain model outputs on PII-related text. Use for research and reproducibility only; handle any residual sensitivity in downstream use responsibly.

Limitations

  • —Results depend on the exact benchmark version, prompt version, and PII-Codex alignment used in the study; different versions may yield different numbers.
  • —Scores are stored as JSON strings for schema consistency across splits; parsing is required for numerical analysis.
  • —Splits differ in scope: pilot_v1 uses n=200 per cell, main_v1 n=2,000; fp16_v1 uses a 300-sample subset; scaling_v1 covers a single 14B model; detector_v1 has no model. Use the split that matches the analysis you are reproducing.

License

See the dataset repository and the linked paper/code repository for license terms. These results are provided for research and reproducibility.

Related Resources