CoolFace
Datasetpublic

timo-pierre-schrader/canonical-order-problem

List Extraction Dataset A dataset for studying how large language models (LLMs) represent and generate multi-valued relations — relations in which a single subject is associated with a set of entities (e.g., "all countries in South America", "all elements of the periodic table"). It accompanies the paper: The Canonical Order Problem: When Large Language Models Are Unreliable Knowledge Bases for Multi-Valued Relations Timo Pierre Schrader, Annemarie Friedrich, Simon Razniewski… See the full description on the dataset page: https://huggingface.co/datasets/timo-pierre-schrader/canonical-order-problem.

sourceHugging Facecc-by-sa-4.0updated 13d agoView on Hugging Face
0likes67downloads
Dataset Card

List Extraction Dataset

A dataset for studying how large language models (LLMs) represent and generate multi-valued relations — relations in which a single subject is associated with a set of entities (e.g., "all countries in South America", "all elements of the periodic table").

It accompanies the paper:

The Canonical Order Problem: When Large Language Models Are Unreliable Knowledge Bases for Multi-Valued Relations Timo Pierre Schrader, Annemarie Friedrich, Simon Razniewski, Lukas Lange

The Canonical Order Problem

LLMs are increasingly used as knowledge bases. While much prior work focuses on extracting single relational triples, most real-world relations are multi-valued and require generating a complete set of entities.

This dataset is built to investigate the canonical order problem: the probabilistic distributions inside LLMs organize many multi-valued relations according to a canonical ordering (e.g., alphabetical, chronological, numerical, or another domain-specific convention). As a result, prompts that deviate from this internal canonical ordering lead to a markedly reduced reliability of LLMs when generating complete sets.

Each example provides a prefix (a subset of the target set) and asks the model to complete the remaining entities. By varying how the prefix is ordered, the dataset isolates the effect of deviating from the canonical order on set-completion reliability.

Topics

The dataset covers 37 distinct multi-valued relations (topics), each with a well-defined, stable number of entities, plus 1 extreme topic used for stress-testing. Topics are categorized by set size:

SizeDefinition
small\S(t)\< 15
regular15 ≤ \S(t)\≤ 50
large\S(t)\> 50
extremevery large sets (used only in shuffled / non_shuffled)

Each topic is assigned a canonical ordering — the most common convention in authoritative sources (e.g., Wikipedia, official lists): alphabetical, chronological, numerical, or other.

Topics span a wide range of domains, e.g. Nobel prize categories, planets of the solar system, German states, NBA teams, U.S. presidents, books of the Bible, constellations, elements of the periodic table, and (for the extreme case) prime numbers up to 100,000.

Dataset Structure

The dataset is organized into 5 configurations (one per ordering strategy), each with train, validation, and test splits.

Note: The train, validation, and test splits contain identical data. They are provided for API compatibility; any single split can be used for evaluation.

Configurations

The configuration name is english_<ordering>, e.g. english_shuffled.

OrderingPrefix construction
shuffledA randomly shuffled subset of the entities (deviates from canonical order)
non_shuffledA sorted subset of the entities (deterministic order)
orderedThe first k entities in the canonical order
one_after_anotherA single entity (each entity used as its own 1-element prefix)

For ordered, one example is generated for each prefix length k = 1 … (n−1). For one_after_another, one example is generated per entity. For shuffled and non_shuffled, a fixed set of random subsets of varying lengths is used per topic.

Row counts (per split)

ConfigurationRows
english_shuffled3,800
english_non_shuffled3,800
english_ordered1,790
english_one_after_another1,827

Features

ColumnTypeDescription
idint16Unique row identifier.
topicstringHuman-readable topic name.
normalized_topic_namestringNormalized topic key.
countint16Total number of entities in the full target set.
list_entitiesSequence[string]All entities of the set, in canonical order.
prefix_entitiesSequence[string]The given prefix (subset of list_entities).
expected_entitiesSequence[string]The remaining entities needed to complete the set (the target).
python_prefixesstringThe prefix formatted as a Python-style list, e.g. ['A', 'B', ...].
bullet_point_prefixesstringThe prefix formatted as a bullet-point list.
prefix_lengthint16Number of entities in the prefix.
list_sizestringSize category: small, regular, large, or extreme.
type_of_orderingstringCanonical ordering: alphabetical, chronological, numerical, or other.
additional_instructionsstringOptional extra prompt instructions (may be null).

The generation task is: given prefix_entities (or its formatted variants), generate expected_entities to complete the full set list_entities.

Citation

If you use this dataset in your research, please cite the paper:

bibtex
TBA

License

This dataset is released under the CC BY-SA 4.0.