Datasetpublic

perplexity-ai/PII-TRACE

PII-TRACE PII-TRACE is a synthetic dataset for privacy-focused named entity recognition (NER) of personally identifiable information (PII), released as a 500-conversation subset of multi-turn dialogues with exact span annotations. Dataset summary This release contains 500 conversations with 2,653 annotated PII spans across nine labels. Among them, 450 conversations contain PII spans and 50 contain none. Data format Each record contains: Field… See the full description on the dataset page: https://huggingface.co/datasets/perplexity-ai/PII-TRACE.

sourceHugging Facemitupdated 2d agoView on Hugging Face

0likes93downloads

Dataset Card

PII-TRACE

PII-TRACE is a synthetic dataset for privacy-focused named entity recognition (NER) of personally identifiable information (PII), released as a 500-conversation subset of multi-turn dialogues with exact span annotations.

Dataset summary

This release contains 500 conversations with 2,653 annotated PII spans across nine labels. Among them, 450 conversations contain PII spans and 50 contain none.

Data format

Each record contains:

Field	Type	Description
`id`	string	Unique record ID.
`turns`	list	Ordered turn objects containing `turn`, `user`, and `assistant`.
`spans`	list	Gold PII spans. An empty list indicates an empty control.

Each item in spans contains:

Field	Type	Description
`label`	string	PII label.
`turn`	integer	Zero-based index of the containing turn.
`source`	string	Message role containing the span (`user` or `assistant`).
`start`	integer	Inclusive Unicode code-point start offset in the selected message.
`end`	integer	Exclusive Unicode code-point end offset in the selected message.
`text`	string	Exact annotated substring.

Do not trim or normalize message text before applying offsets.

Labels

Label	Description
`private_person`	Person name
`private_date`	Date
`private_url`	Link or route
`private_address`	Mailing or location address
`account_number`	Account or reference number
`private_email`	Email address
`private_phone`	Telephone number
`other_pii`	Other identifier
`secret`	Access code or credential-like value

Evaluation

Supply turns to the model and keep spans hidden. Report micro and macro exact typed-span precision, recall, and F1 over (turn, source, start, end, label). Untyped exact-span F1 over (turn, source, start, end) may also be reported.

This subset may be used for training, evaluation, and error analysis in PII span detection research.

Limitations

This is a controlled benchmark rather than an estimate of real-world PII prevalence or deployment-level performance. Its English conversations and identifier formats are more regular than production traffic, and current gold spans occur in user messages. The secret label covers credential-like sensitive values but does not assert that every such value is statutory PII.

License

MIT.

Citation

If you use or reference this work, please cite:

bibtex

@article{zhang2026piitrace,
  title = {{PII-TRACE}: A Benchmark for Context-Aware {PII} Detection in Multi-Turn {LLM} Conversations},
  author = {Zhang, Kaiyuan and Wang, Chuan and Zhong, Joey and Fryzel, Paul and Polley, Kyle and Ma, Jerry and Li, Ninghui},
  journal = {arXiv preprint arXiv:2609.22200},
  year = {2026},
  url = {https://arxiv.org/abs/2609.22200}
}