tax-free/scientific-ideation-diversity

Scientific Ideation Diversity

Raw data release for the paper "On the Effects of Reasoning Effort and Prompt-Based Diversification on Scientific Ideation Diversity" (Discovery Science 2026).

The paper studies how reasoning effort (low/medium/high) and prompt-based diversification (Verbalized Sampling, String Seed of Thought) shift the diversity of LLM-generated scientific ideas across three frontier models — Claude Sonnet 4.6, GPT-5.4, Gemini 3.1 Pro — evaluated with lexical metrics, three embedding models, and pairwise + per-idea LLM judges (about 1,400,000 pairwise judgments).

Code, prompts, and reproduction scripts: <https://github.com/bioinfo-tsukuba/scientific-ideation-diversity> Every figure, table, and reported statistic in the paper can be rebuilt from this dataset with the scripts in that repository (see its REPRODUCING.md; start with tools/materialize_hf_dataset.py, which unpacks this dataset into the directory layout the scripts expect).

Design principle: raw files only

This dataset intentionally contains no aggregated or intermediate tables. It holds the byte-unmodified raw pipeline outputs (JSONL) plus embeddings, and everything else is derived by the published scripts. The only two transformations applied at packaging time:

1. 1.Deduplication — each run's samples.jsonl existed in several embedder subdirectories; one copy is kept.
2. 2.Embeddings float64 → float32 — the APIs deliver less than float32 precision, so this halves the size with no loss relevant to any reported metric (which use 2–3 decimals).

Layout

ideas/
  effort-axis/{claude-sonnet-4-6,gpt-5-4,gemini-3-1-pro}.jsonl.gz
  prompt-axis/{claude,gpt54,gemini31pro}_{vs,ssot}_{low,high}.jsonl.gz
judge/
  diversity/...   pairwise judgments (four ordered labels A–D), per generator × judge
  quality/...     per-idea originality/feasibility/clarity (1–10) + judge analysis text
errors/           generation/judging error logs (basis of the ≥99.96% coverage figure)
embeddings/
  {effort-axis,prompt-axis}/<run>/<embedder>/embeddings{,_purpose,_mechanism,_evaluation}.npy

• —Ideas (samples.jsonl records): one line per generated idea. Self-describing fields include idea_model, keyword, category, prompt_style, effort, sample_index, the structured {purpose, mechanism, evaluation} facets, the combined idea text, the full raw API response, and token usage. 30 ideas per keyword; keyword set is LiveIdeaBench's 1,180 keywords (Claude Sonnet 4.6: 1,160 — 20 biosecurity-adjacent keywords refused). Effort tiers on the effort axis: none/low/medium/high for Claude and GPT-5.4, low/medium/high for Gemini (the paper reports low/medium/high).
• —Judges: GPT-4.1 and Claude Haiku 4.5 (both axes), plus Claude Sonnet 4.6 / GPT-5.4 / Gemini 3.1 Pro as quality judges on the Q₁∪Q₄ keyword subsets. Judge records embed the judge model and prompt-template version (liveideabench_critic_prompt@eb9d0abf, fluency_critic_prompt@ed050700).
• —Embedders: OpenAI text-embedding-3-large (3072-d), Amazon titan-embed-text-v2:0 (1024-d), SPECTER2 base and adhoc_query adapter (768-d). Row order matches the line order of the corresponding samples.jsonl; _purpose/_mechanism/_evaluation are per-facet embeddings, the unsuffixed file embeds the combined text.

Generation provenance

• —Generation models: jp.anthropic.claude-sonnet-4-6 (Bedrock Converse), gpt-5.4-2026-03-05 (OpenAI Responses), gemini-3.1-pro-preview (Vertex AI); vendor-default sampling parameters, only the reasoning-effort level varied; schema-constrained structured output; seed 42 for keyword subsampling.
• —Keywords and judge prompts come from LiveIdeaBench (<https://github.com/x66ccff/liveideabench>, MIT License, Copyright (c) 2025 Kai Ruan), pinned at commit fc95111c7920cb4ca02e27f5ab9730f3886ca07c; utils/prompts.json sha256 8aa2d481…cbe81e11d, keyword CSVs pinned in the code repository's data/benchmarks/liveideabench_manifest.json. The generation prompts extend LiveIdeaBench's single-keyword prompt with the {purpose, mechanism, evaluation} facet schema of Scideator.

License and attribution

• —Data: CC BY 4.0. Please cite the paper below.
• —Keyword lists and judge prompts are derived from LiveIdeaBench (MIT); the MIT notice is reproduced in the code repository's NOTICE.md.
• —Generated idea texts are model outputs from Anthropic, OpenAI, and Google models; judge outputs from OpenAI and Anthropic models; embeddings from OpenAI, Amazon, and AllenAI (SPECTER2) models.

Citation

@inproceedings{chinen2026ideationdiversity,
  title     = {On the Effects of Reasoning Effort and Prompt-Based Diversification on Scientific Ideation Diversity},
  author    = {Chinen, Yu and Ozaki, Haruka},
  booktitle = {Discovery Science: 29th International Conference, DS 2026, Mainz, Germany},
  series    = {Lecture Notes in Artificial Intelligence},
  publisher = {Springer},
  year      = {2026},
}