LevenKoko/MIRAGE-CanaryDocs

MIRAGE CanaryDocs

MIRAGE CanaryDocs is an English synthetic enterprise-document dataset for structured privacy-unit, canary, and ordered multi-chunk evaluation. It is the companion dataset for the EMNLP 2026 paper When Metadata Remembers: Ordered Provenance Enables Document-Level Embedding Inversion.

Project documentation and schemas are also available in the MIRAGE GitHub repository.

Dataset summary

The dataset contains complete synthetic documents, ordered token chunks, document-level Gold units, exact surface occurrences, occurrence-to-chunk alignments, and cross-location dependency relations. It is designed for controlled research on document reconstruction, structured privacy-unit recovery, canary recovery, ordered provenance, long-context retrieval, and parser evaluation.

All three splits intentionally publish their complete inputs and Gold annotations. The test split is an openly labeled reference split, not a hidden test set or evaluation-server split.

The documents cover 24 scenario families across account, calendar, approval, location, finance, and release workflows. Shadow contains 12 families; Dev and Test contain all 24 families.

Loading the canonical layers

Each normalized entity layer is exposed as a separate Hugging Face configuration because the layers have different schemas. Every configuration provides shadow, dev, and test splits.

from datasets import load_dataset

documents = load_dataset("LevenKoko/MIRAGE-CanaryDocs", "documents")
units = load_dataset("LevenKoko/MIRAGE-CanaryDocs", "units")
chunks = load_dataset("LevenKoko/MIRAGE-CanaryDocs", "chunks")

print(documents["test"][0]["title"])
print(units["test"][0]["type"])
print(chunks["test"][0]["chunk_index"])

Available configurations:

Repository layout

The normalized JSONL files under data/ are the canonical source of record. Other formats are materialized convenience representations of the same public release.

data/
  documents/{shadow,dev,test}.jsonl
  sections/{shadow,dev,test}.jsonl
  paragraphs/{shadow,dev,test}.jsonl
  chunks/{shadow,dev,test}.jsonl
  units/{shadow,dev,test}.jsonl
  occurrences/{shadow,dev,test}.jsonl
  alignments/{shadow,dev,test}.jsonl
  dependencies/{shadow,dev,test}.jsonl
views/
  bundled/{shadow,dev,test}.jsonl
  canaries/{shadow,dev,test}.jsonl
  benchmark/{dev,test}.parquet
database/canarydocs.sqlite
metadata/

• —views/bundled/: one self-contained nested document per JSONL row.
• —views/canaries/: canary-only units with occurrence and chunk mappings.
• —views/benchmark/: nested Dev/Test Parquet views.
• —database/canarydocs.sqlite: relational mirror containing only public release tables and views.
• —metadata/: JSON Schemas, taxonomy, family definitions, statistics, split metadata, chunking semantics, file inventory, and SHA-256 checksums.

To load a bundled view directly:

from datasets import load_dataset

bundled_test = load_dataset(
    "json",
    data_files={
        "test": "https://huggingface.co/datasets/LevenKoko/MIRAGE-CanaryDocs/resolve/main/views/bundled/test.jsonl"
    },
)
print(bundled_test["test"][0]["gold_units"][0])

Data model

document -> section -> paragraph
document -> chunk
document -> unit -> occurrence <-> chunk
                         |
                         +-> dependency -> occurrence

• —A unit is one document-level Gold value.
• —An occurrence is one exact realization of a unit in document text.
• —An alignment records whether a chunk fully contains or intersects an occurrence.
• —A dependency links consecutive occurrences of a repeated unit and records whether the relation crosses chunk or non-overlapping chunk boundaries.

Stable identifiers include document_id, section_id, paragraph_id, chunk_id, unit_id, occurrence_id, alignment_id, and relation_id. Character and token spans are zero-based, half-open intervals.

Annotation taxonomy

Gold units use nine mutually exclusive types:

• —entity_mention
• —email_account
• —date_time
• —amount_numeric
• —url_path_token
• —identifier_case_id
• —canary_string
• —rare_domain_ngram
• —other_non_anchorable

Canaries use five surface-format families: account, identifier, other, path, and URL. The 33,379 canary values are unique in this release.

Chunking

Chunk boundaries use a frozen intfloat/e5-large-v2 tokenizer configuration without special tokens, padding, or truncation. The window size is 128 tokens.

Exact tokenizer metadata and hashes are recorded in metadata/chunking.json.

SQLite access

Download database/canarydocs.sqlite and query the relational mirror directly:

SELECT d.document_id, u.surface, c.chunk_index
FROM units u
JOIN occurrences o USING (unit_id)
JOIN alignments a USING (occurrence_id)
JOIN chunks c USING (chunk_id)
JOIN documents d USING (document_id)
WHERE u.is_canary = 1
ORDER BY d.document_id, c.chunk_index;

Integrity and release metadata

• —Release version: 1.0.0
• —Release date: 2026-08-30
• —Canonical format: normalized JSONL
• —Relational mirror: SQLite with foreign keys
• —Per-file hashes: metadata/checksums.sha256
• —Machine-readable file inventory: metadata/file_inventory.json
• —JSON Schemas: metadata/schemas/

After downloading the complete repository, verify it with:

sha256sum --quiet -c metadata/checksums.sha256

Intended uses

• —Structured privacy-unit and canary recovery evaluation.
• —Ordered-context and document-level reconstruction analysis.
• —Chunk/document alignment and long-context retrieval research.
• —Synthetic benchmark development and parser validation.

Limitations

• —The text is synthetic and may contain formulaic phrasing or intentionally unusual identifiers.
• —The dataset should not be treated as evidence about real people, organizations, credentials, or operational systems.
• —All hosted URLs and email domains use non-routable .invalid namespaces.
• —Because all Gold annotations are public, reported Test results must be interpreted as evaluation on an open reference set rather than a blind benchmark.
• —Canary strings are public benchmark markers, not secret production credentials or live leak probes.

Citation

If you use MIRAGE CanaryDocs, please cite the accompanying EMNLP 2026 paper:

@inproceedings{zheng2026metadata,
  title     = {When Metadata Remembers: Ordered Provenance Enables Document-Level Embedding Inversion},
  author    = {Zheng, Liwen and Li, Qing and Zou, Qingsong and Jiang, Yong},
  booktitle = {Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing},
  year      = {2026}
}

Structured citation metadata is provided in CITATION.cff.

License

MIRAGE CanaryDocs is released under the Creative Commons Attribution 4.0 International license (CC BY 4.0). See LICENSE for details.