anon-neurips26-3390/IndustryBench

IndustryBench: Probing the Industrial Knowledge Boundaries of LLMs

Anonymized review copy — NeurIPS 2026 Evaluations & Datasets Track, Submission 3390 (under review).

IndustryBench is a benchmark for evaluating the industrial procurement knowledge of large language models. It comprises 2,049 QA pairs grounded in Chinese national standards (GB/T) and structured industrial product records, with item-aligned renderings in Chinese, English, Russian, and Vietnamese.

Erratum note: Figure 1 of the submission misprints "2048 QA pairs"; the correct count is 2,049, as stated in the abstract and §3 and as contained in this release (2,049 rows).

Overview

The distributions follow the natural frequency of the verified source pool; low-support slices (Fault Diagnosis, Engineering Calculation) should be interpreted as diagnostic signals rather than precise rankings.

Dataset fields

Verifying this release

Note on downloading: huggingface_dataset.csv is stored via Git LFS. If you git clone without git-lfs installed, you will get a 133-byte pointer file instead of the data — either install git-lfs first, or download the file directly from .../resolve/main/huggingface_dataset.csv (or use datasets.load_dataset). The two verification scripts below are pure standard library; evaluate.py additionally requires requests (see requirements.txt).

• —python scripts/verify_marginals.py — asserts the row count (2,049), completeness of all four language versions and knowledge_text (no empty cells), and the exact capability/difficulty/industry marginal counts listed above; prints the dataset file's SHA-256 (the full per-file list is in checksums.txt).
• —python scripts/inspect_random_items.py --seed 3390 -n 20 — prints a seeded random sample of items with question, reference answer, and knowledge_text side by side, for direct inspection of item quality and label plausibility against the source excerpt.

Evaluation protocol

evaluate.py implements the full evaluation pipeline described in §4 of the submission: multi-language answering prompts (embedded, zh/en/ru/vi), LLM-as-Judge raw scoring on the 0–3 rubric (JUDGE_PROMPT_TEMPLATE_ZH / JUDGE_PROMPT_TEMPLATE_EN, embedded verbatim from Appendix I and routed by language — Chinese prompt for the Chinese benchmark, English prompt for the translations), and the per-item safety-violation review against knowledge_text (SAFETY_REVIEW_PROMPT_TEMPLATE, embedded verbatim from Appendix J), with SV-flagged responses set to 0. It targets any OpenAI-compatible API:

python3 evaluate.py --language zh  --api-base YOUR_API_BASE --api-key $YOUR_KEY \
    --model MODEL_NAME --judge-model JUDGE_MODEL_NAME
python3 evaluate.py --language all --api-base YOUR_API_BASE --api-key $YOUR_KEY \
    --model MODEL_NAME --judge-model JUDGE_MODEL_NAME

Pass --judge-model explicitly: the submission uses one fixed, human-calibrated judge for every evaluated system, whereas the script would otherwise let the evaluated model judge itself. Results are written per language to results/{model}_{lang}_results_{timestamp}.csv, and runs resume from a per-language checkpoint. Following the protocol, a successful API call that returns an empty response is scored 0 rather than dropped; genuine API failures are retried and reported separately in the run summary.

The remaining pipeline prompts (Stage-4 search-query generation, translation, translation faithfulness review) are provided under prompts/; see prompts/README.md for provenance.

Intended use and limitations

This dataset is an evaluation resource for diagnosing industrial procurement knowledge in LLMs (capability/industry/difficulty slices, multilingual stability, safety-violation behavior). It is grounded exclusively in Chinese national standards (GB/T) and Chinese industrial e-commerce product records; ISO/DIN/ANSI regimes are not represented. High scores do not certify real-world safety, legal compliance, or procurement readiness. Do not use this dataset for training or fine-tuning; doing so undermines its validity as an evaluation resource. Machine-readable metadata, including Responsible-AI fields, is provided in croissant.json.

Errata, error reports, and versioning

This release is version 1.0.0 (matching the version field in croissant.json). Errors or defective items can be reported through this repository's Community/Discussions page; corrections are released under the versioning scheme described in Appendix T (ds-7) of the submission, with all changes tracked. Known erratum: the submission's Figure 1 misprints "2048 QA pairs" (correct count: 2,049; see the note at the top of this README).

Licensing

Two layers: (1) the QA pairs, labels, prompts, and code in this repository are released under the MIT license (see LICENSE) — the repository-level license: mit tag covers this original material only; (2) the short source excerpts in knowledge_text (GB/T standards and product records) are included as quotations for evaluation and verification purposes and are not relicensed — GB/T excerpts retain their standard identifiers in-text; the release contains no full standard documents and no raw product pages.

Citation

Anonymous authors, IndustryBench: Probing the Industrial Knowledge Boundaries of LLMs, under review at the NeurIPS 2026 Evaluations & Datasets Track (Submission 3390). Citation information will be added upon de-anonymization.