Meerkat-AI/triz-gold-benchmark

🇺🇸 English | 🇨🇳 中文

triz-gold-benchmark

A Chinese TRIZ (Theory of Inventive Problem Solving) evaluation benchmark, companion to Meerkat-TRIZ-v1 and the meerkat-triz evaluation harness.

Contents

One JSON object per line:

{"id": "v5_gold_000", "subset": "ariz_guidance", "question": "...", "keywords": ["..."]}

Six task subsets (v4 / v5 item counts):

Important: this public release contains no reference answers

The questions and expected keywords were LLM-generated from third-party TRIZ textbooks and course materials; the reference answers are derivative rewrites of copyrighted content. To control copyright risk, this public release only publishes question + keywords + subset — no reference_answer field.

Implications:

• —The keyword track is fully reproducible (the harness keyword track only needs question/keywords).
• —The judge track loses its reference-answer anchor: in the judge prompt the reference answer is the quality anchor; without it, absolute scores are not directly comparable to the official reports (paired diffs are less affected).
• —For the full version for academic research, please contact the authors via a repo issue.

Usage

meerkat-eval --config configs/eval_v5.json \
    --adapter-path <adapter> --tag my_run \
    --eval-file triz_gold_v5_public.jsonl \
    --baseline-results <base result.json>

Leakage statement

This benchmark never entered the training distribution of Meerkat-TRIZ-v1: a 3-gram Jaccard ≥0.5 scan of the training set against both gold sets found 0 hits.

License

CC-BY-NC-4.0 (non-commercial research use). The question texts are derived from third-party copyrighted TRIZ materials; contact the authors before any commercial use.