CoolFace
Modelpublic

Parssky/industrial-instruction-qwen4b-claude

sourceHugging Faceotherupdated 1mo agoView on Hugging Face
0likes69downloads
Model Card

Industrial-Instruction Qwen3-4B (Claude-generated data)

Qwen3-4B-Instruct-2507 fully fine-tuned on the `panasonic_qa_claude_v1` config of Industrial-Instruction — the split generated with Claude-Opus-4.6 using an identical pipeline, prompts and scenario definitions.

A companion model trained on the open-weight-generated split is available at Parssky/industrial-instruction-qwen4b.

Paper

Industrial-Instruction: An End-to-End Framework for Building Instruction-Tuning and Benchmark Datasets from Industrial Technical Reports

Base Model

Qwen3-4B-Instruct-2507

Results

Panasonic benchmark (panasonic_qa_claude_v1 test split, 1,000 items):

ConfigurationF1JaccardSet-Match Acc.
Base, with RAG58.55%54.00%40.90%
Fine-tuned, with RAG72.66%68.88%56.40%
Base, no RAG59.24%54.71%41.60%
Fine-tuned, no RAG72.72%68.93%56.40%

General knowledge (MMLU): 72.13% base → 72.08% fine-tuned (−0.05) — essentially no forgetting. Several subjects improve: globalfacts (+8.00), collegemathematics (+7.00), machinelearning (+5.36), securitystudies (+4.49), international_law (+3.30).

FailureSensorIQ: AccOrgIBM 34.0% → 49.6%, F1-Macro 40.0% → 33.5%, F1-Micro 66.0% → 50.3%.

Note: the two models are evaluated on different held-out test splits, each generated by its own generator model. The baseline scores therefore differ (28.5% vs. 40.9%), and the two improvement magnitudes are indicative rather than a controlled head-to-head comparison.

Limitations

  • —Accuracy on perturbed (rephrased) FailureSensorIQ questions is 0%, both before and after fine-tuning. The training data contains no paraphrased question variants, so this model should not be relied on where question phrasing varies.
  • —Fine-tuning costs 1.26 points of MMLU accuracy, concentrated in Humanities (moral-reasoning subjects drop ~10 points).
  • —Source documentation comes from a single manufacturer; terminology may not transfer.

Intended Use

Research and benchmarking for industrial retrieval-augmented generation (RAG), evidence integration, and technical-domain QA.

Notes

  • —Inference-ready weights and tokenizer files only; training checkpoints and optimizer states are intentionally excluded.

Source Code

GitHub repository: https://github.com/parssky/industrial-instruction