schonsense/human_ai_story_contrastive_v4

Human–AI Story Contrastive Six-Source Dataset

Dataset summary

This dataset contains 1,443 matched short-story prompt groups with human and machine-written realizations drawn from six source populations.

The dataset is organized one row per `group_id` rather than one row per text. Each row preserves the common writing prompt, the human reference story, the available earlier machine generations, the starting-policy generation, and two GPT-5.6 Sol fields added for frontier-model comparison.

Across all source fields, the dataset contains 8,619 texts:

[image]

The two GPT-5.6 Sol fields are intentionally different:

• —`gpt56_blind` is a natural-policy generation from the writing prompt.
• —`gpt56_conditioned` is a paired transformation probe: GPT-5.6 Sol is given the human story and asked to rewrite it from scratch at the sentence level while preserving its narrative content, including the same events, entities, causal relationships, point of view, tense, outcome, and approximate length.

`gpt56_conditioned` should not be treated as a natural-policy Sol sample. Its purpose is to expose stylistic territory reachable by the same frontier model when supplied the paired human realization.

Why this dataset exists

The dataset was assembled for research into expressive-support geometry: how much stylistic territory different language-model policies occupy, how concentrated or diffuse that territory is, and how human and machine populations differ when prompt/content variation is controlled as much as possible.

The human corpus is best treated as a reference distribution for plausible expressive breadth, not as a single target centroid.

Because the data are grouped by prompt/story identity, the dataset supports both:

• —population-level comparisons between source distributions, and
• —matched within-group comparisons between different realizations of the same underlying prompt.

Dataset structure

The canonical file is:

dataset.jsonl

Each JSONL row corresponds to one group_id.

Top-level fields

GPT-5.6 Sol nested records

Both gpt56_blind and gpt56_conditioned contain:

The join-only hashes used to validate the Sol files against the canonical base rows are verified during dataset construction rather than duplicated inside the nested Sol objects.

Source semantics

Human

human_response is the human-authored reference realization for the matched writing prompt.

GPT-3.5 and Opus

Earlier off-policy machine generations are retained inside off_policy_responses. These sources are not complete for all 1,443 groups, which is why their counts are slightly smaller than the four complete source populations.

pi0

starting_policy_response is the starting-policy generation used in the associated expressive-deconcentration experiments. It should not be interpreted as a generic base-model population: it comes from a narrative/RP-oriented local policy and therefore has its own training history and stylistic biases.

GPT-5.6 Sol blind

gpt56_blind.text is generated from the writing prompt without access to the paired human story. It is the appropriate field when treating GPT-5.6 Sol as a natural machine-policy population in this dataset.

GPT-5.6 Sol conditioned

gpt56_conditioned.text is produced by asking GPT-5.6 Sol to rewrite the paired human story while preserving its narrative content and realizing the same story in its own prose.

This field is intended for questions such as:

• —What changes when a strong model is supplied a human realization of the same story?
• —Which stylistic regions are reachable under strong content-preserving conditioning?
• —How does the conditioned transformation differ from the remaining machine-to-human residual?

It is not intended to represent ordinary GPT-5.6 Sol generation behavior.

Style-geometry overview

The figures below use full-dimensional StyleDistance story embeddings. They are included as descriptive analyses of the source populations, not as ground-truth measures of writing quality or "human-likeness."

Population spread

[image]

Covariance trace summarizes total population spread in the analysis representation. In this dataset, the human population has the largest total spread. GPT-5.6 Sol conditioned has substantially greater spread than GPT-5.6 Sol blind.

Covariance effective rank

[image]

Effective rank describes how broadly covariance energy is distributed across dimensions. It should not be treated as interchangeable with total spread.

A notable result is that GPT-5.6 Sol blind has relatively modest total spread but high covariance effective rank, whereas GPT-5.6 Sol conditioned has much larger total spread but a more concentrated spectral shape.

This illustrates why "breadth" should not be reduced to a single scalar: location, total radius/spread, and spectral concentration are different properties.

Matched Sol conditioning geometry

[image]

For matched groups, the conditioned Sol population is closer to the human realization than blind Sol is, while the blind-to-conditioned transformation itself remains substantial.

This should not be interpreted as evidence for one universal "humanward direction." The transformation is high-dimensional, and the conditioning movement only partially explains the remaining conditioned-to-human residual.

Example access

import json

with open("dataset.jsonl", "r", encoding="utf-8") as f:
    row = json.loads(next(f))

prompt = row["source_prompt"]
human = row["human_response"]
pi0 = row["starting_policy_response"]
sol_blind = row["gpt56_blind"]["text"]
sol_conditioned = row["gpt56_conditioned"]["text"]

gpt35 = row["off_policy_responses"].get("story_gpt_3_5", {}).get("text")
opus = row["off_policy_responses"].get("story_opus", {}).get("text")

With 🤗 Datasets:

from datasets import load_dataset

ds = load_dataset(
    "json",
    data_files="dataset.jsonl",
    split="train",
)

Intended uses

This dataset is suitable for research on:

• —human/machine stylistic distribution differences;
• —paired prompt-conditioned writing comparisons;
• —expressive-support and mode-concentration analysis;
• —style-space representation learning;
• —reward-model or critic development;
• —machine-generation detection research;
• —preference and policy-optimization diagnostics;
• —comparison of natural-policy generation with content-preserving conditioned transformation;
• —evaluation of whether training interventions broaden or merely shift a model's output distribution.

The grouped structure is particularly useful when the research question requires separating prompt/content effects from source/model effects.

Uses requiring care

Do not mix blind and conditioned Sol as if they were equivalent samples

gpt56_blind is a natural-policy population.

gpt56_conditioned is a human-conditioned transformation population.

Pooling them without preserving that distinction changes the meaning of the dataset.

Do not treat one embedding metric as a complete quality measure

The included StyleDistance analyses describe geometry in one frozen representation. They are useful for comparative distributional analysis but do not establish writing quality, factuality, coherence, or human preference on their own.

Content and style are not perfectly separable

Matched prompts and conditioned rewrites substantially improve control over content, but no text representation perfectly isolates style from semantics, narrative structure, length, or model-specific realization choices.

Dataset limitations

• —The dataset focuses on English short-story generation and should not be assumed to generalize directly to other languages or task domains.
• —GPT-3.5 and Opus are slightly incomplete relative to the 1,443-group canonical set.
• —The source populations come from different models and generation contexts, so model identity and generation procedure remain part of the observed distribution.
• —gpt56_conditioned depends directly on the paired human story and therefore cannot be used as an independent natural-policy sample.
• —The pi0 population comes from a narrative/RP-oriented policy and should not be treated as representative of machine-generated prose in general.
• —Style-geometry statistics are representation-dependent.
• —The dataset is designed for comparative research, not as a universal ranking of model writing quality.

Integrity and provenance

The canonical six-source aggregate contains:

1,443 rows
1,443 unique group_ids
0 blank gpt56_blind texts
0 blank gpt56_conditioned texts

The construction pipeline validates:

• —exact group_id agreement between the base, blind-Sol, and conditioned-Sol files;
• —source-prompt text agreement;
• —source-prompt SHA-256 agreement;
• —human-response SHA-256 agreement;
• —consistency of the paired human hash across the blind and conditioned Sol records;
• —uniqueness of all joined group IDs.

The accompanying manifest.json records the construction inputs, source counts, and validation status.

Repository layout

A recommended repository layout is:

.
├── README.md
├── dataset.jsonl
├── manifest.json
└── figures/
    ├── 01_dataset_source_counts.png
    ├── 02_population_covariance_trace.png
    ├── 03_population_effective_rank.png
    └── 04_sol_matched_distances.png

Citation

If you use this dataset, please cite the Hugging Face repository and any associated project/report linked from the repository once available.