latentmd-neurips26/LatentMD

LatentMD

Benchmarking Markdown Boundary Failures in LLM-Generated Text — NeurIPS 2026 E&D Track submission.

This repository hosts the dataset artifact for LatentMD: the 4,179 prompts that constitute the benchmark, ~37,000 reference responses from 9 frontier models, and illustrative output samples. The accompanying evaluation code (CLI, metric definitions, statistical tests) lives in a separate code repository on GitHub under MIT.

What LatentMD measures

LLM Markdown output is usually judged on whether it renders well and says the right thing. LatentMD adds a third axis: does the fence structure parse correctly under spec-strict CommonMark? When it does not, the response can look fine in lenient chat UIs while silently corrupting any downstream consumer that depends on a real parser — agents, code extractors, RAG indexers, structured-output pipelines.

The benchmark separates boundary correctness (does the Markdown parse?) from content correctness (does it satisfy the request?), then reports the cell that prior benchmarks systematically score as "passing":

Latent failure — outputs that are content-correct but boundary-broken.

Headline result from the paper: 38.0% of valid main-grid LLM generations are content-correct but boundary-broken. The rate is robust across model size, language, and prompt-axis condition.

Dataset layout

LatentMD-dataset/
├── prompts/
│   └── priority/                            # 4,179 deterministic prompts
│       ├── main_prompts.yaml                # 2,160  (RQ1 + RQ2 main grid)
│       ├── delimiter_family_prompts.yaml    #   288  (D-axis: tilde vs backtick)
│       ├── delimiter_length_prompts.yaml    #   432  (D-axis: outer fence length)
│       ├── delimiter_inner_prompts.yaml     #   432  (D-axis: inner run requirement)
│       ├── cross_family_prompts.yaml        #   288  (D-axis: cross-family nesting)
│       ├── hint_prompts.yaml                #   432  (mitigation ablation)
│       ├── inline_prompts.yaml              #    27  (RQ4: inline code spans)
│       ├── crossformat_prompts.yaml         #    90  (Python / JSON appendix)
│       ├── natural_prompts.yaml             #    30  (external-validity natural prompts)
│       ├── stability_subset_prompts.yaml    #    30  (separate stability subset)
│       └── manifest.yaml                    # split metadata
├── responses/
│   ├── main_grid/<model>.jsonl              # 9 models × 2,160 main prompts
│   └── ablation/<model>.jsonl               # 9 models × ablation prompts
├── samples/                                 # 12 illustrative GPT-4o outputs (one per A×B cell)
├── examples/                                # Smoke-test bundle for the evaluation CLI:
│   ├── sample_responses.jsonl               #   5-record GPT-4o slice of the main grid
│   ├── expected_output.txt                  #   what `scripts/run_evaluation.py --responses ...` should print
│   └── README.md                            #   how to run it
├── LICENSE                                  # CC BY-SA 4.0
├── croissant.json                           # Croissant 1.0 + RAI metadata
└── README.md                                # this file

stability_subset_prompts.yaml (30 records) is an appendix-only subset for the stability analysis and is not counted toward the 4,179.

Schema

Prompt YAML records

Each entry in prompts/priority/*.yaml is a YAML mapping with at least:

Response JSONL records (responses/.../<model>.jsonl)

The same JSONL schema is what the evaluation CLI accepts via python scripts/run_evaluation.py --responses <file>.

Loading

With Hugging Face datasets

from datasets import load_dataset

# Load main-grid prompts
prompts = load_dataset("latentmd-neurips26/LatentMD", data_files="prompts/priority/main_prompts.yaml")

# Or load a particular model's reference responses
responses = load_dataset("latentmd-neurips26/LatentMD",
                        data_files="responses/main_grid/gpt_4o.jsonl",
                        split="train")

As raw files

git clone https://huggingface.co/datasets/latentmd-neurips26/LatentMD
ls LatentMD/prompts/priority/

Evaluating your own model

The evaluation pipeline is hosted in the code repo (MIT). Three steps:

1. 1.Generate responses with your model on the prompts in this dataset, in the JSONL schema above.
2. 2.Clone the code repo and place this dataset under its data/ directory:

   git clone https://anonymous.4open.science/r/LatentMD-code-B51D LatentMD-code
   cd LatentMD-code
   git clone https://huggingface.co/datasets/latentmd-neurips26/LatentMD data

1. 1.Run the evaluator:

   python scripts/run_evaluation.py --responses my_responses.jsonl --output-dir ./out/

The console summary prints the four-cell decomposition (Both correct / Latent failure / Format-only / Both wrong) and a per-record JSONL goes to out/per_record/.

A 5-record smoke-test sample is bundled in the code repo at examples/sample_responses.jsonl.

Provenance

• —TASK slot text: mechanically extracted from McEval (Chai et al., ICLR 2025) instruction fields, stratified by difficulty (5 easy + 5 middle + 5 hard per language; seed=42). McEval is not redistributed here; only per-prompt task text is included via the slot-filling pipeline.
• —LANG slot: 9 programming languages selected by 2-of-3 consensus across TIOBE Index (Apr 2025), GitHub Octoverse 2024, Stack Overflow Developer Survey 2025; plus 3 structured formats (HTML, JSON, Markdown) drawn from McEval's non-programming coverage set.
• —Model responses: collected with temperature = 0 (greedy decoding), max_tokens = 7168. Reasoning modes disabled. Provider-specific hyperparameters are documented in the paper appendix.

Changelog

• —v1.0.0 (2026-05-07): Initial release for NeurIPS 2026 E&D Track submission.

License

This dataset is released under Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0).

The CC BY-SA 4.0 license matches McEval's data license: the McEval-derived slots.TASK text inherits McEval's ShareAlike obligation, and distributing the entire dataset under CC-BY-SA-4.0 keeps the artifact internally consistent. McEval itself is not redistributed in this repository.

The associated evaluation code is released separately under the MIT License in the code repository on GitHub.

Citation

@inproceedings{anonymous2026latentmd,
  title     = {{LatentMD}: Benchmarking {Markdown} Boundary Failures in {LLM}-Generated Text},
  author    = {Anonymous Authors},
  booktitle = {Submitted to NeurIPS 2026 Evaluations and Datasets Track},
  year      = {2026},
  note      = {Under double-blind review}
}