Modelpublic

yuk1chan/qwen3-4b-structeval-yamlxml-boost-v2-lr6e-6

sourceHugging Faceapache-2.0updated 7mo agoView on Hugging Face

0likes6downloads

Model Card

qwen3-4b-structeval-yamlxml-boost-v2-lr6e-6

This repository provides a LoRA adapter fine-tuned from Qwen/Qwen3-4B-Instruct-2507 using QLoRA (4-bit, Unsloth).

Training Objective

This adapter is trained to improve structured output accuracy (JSON / YAML / XML / TOML / CSV).

Strategy 2 Revised: YAML/XML Clean Expansion 🔥

Training Configuration

—Base model: Qwen/Qwen3-4B-Instruct-2507
—Method: QLoRA (4-bit)
—Max sequence length: 512
—Epochs: 1
—Learning rate: 6e-06
—LoRA: r=16, alpha=32

Dataset: Cleaned StructEval + YAML/XML Boost (Revised)

Strategy 2 Revised: Three-Pronged Approach

Based on expert AI analysis, this revision addresses the root causes of YAML/XML failures:

1.u-10bei系: 「Output:」以降だけを抽出
2.Problem: u-10bei datasets contain "Approach: → Output:" structure
3.Solution: Extract only content after "Output:" marker
4.Effect: Removes "explanation before output" tendency

1.daichira系: 2倍ブースト（「Return ONLY」パターン）
2.Problem: Model needs more "Return ONLY YAML/XML" examples
3.Solution: Boost daichira datasets by 2x
4.Effect: Strengthens "direct output without explanation" pattern

1.YAML/XML: 2倍ブースト
2.Problem: YAML/XML are the weakest formats (94.3%, 85.0%)
3.Solution: Boost YAML/XML samples by 2x
4.Effect: More training data for weak formats

Data Cleaning Pipeline:

1.CoT tags removal: <thinking>...</thinking> completely removed
2.Code fence removal: ```yaml, ``json, ``xml, ``toml, ```csv removed
3.Leading phrase removal: "Here's the output:", "Sure!", etc. removed
4.🔥 Output extraction: For u-10bei datasets, extract only content after "Output:" marker
5.Format validation: JSON/YAML/XML/TOML/CSV parsing validation
6.Deduplication: Exact duplicates removed

Format Distribution (Estimated):

—YAML: ~12,000-15,000 (40-50%) 🔥 2x boost
—XML: ~6,000-8,000 (25-30%) 🔥 2x boost
—JSON: ~4,000-5,000 (15-20%)
—TOML: ~2,500-3,000 (10%)
—CSV: ~2,500-3,000 (10%)

Total: ~25,000-30,000 samples

Source Datasets:

u-10bei系（「Output:」抽出適用）:

—u-10bei/structureddatawithcotdataset512v2
—u-10bei/structureddatawithcotdataset512v4
—u-10bei/structureddatawithcotdataset512v5
—u-10bei/structureddatawithcotdataset_512
—u-10bei/structureddatawithcotdataset_v2
—u-10bei/structureddatawithcotdataset

daichira系（2倍ブースト）:

—daichira/structured-3k-mix-sft 🔥 2x
—daichira/structured-5k-mix-sft 🔥 2x
—daichira/structured-hard-sft-4k 🔥 2x

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base = "Qwen/Qwen3-4B-Instruct-2507"
adapter = "yuk1chan/qwen3-4b-structeval-yamlxml-boost-v2-lr6e-6"

tokenizer = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(
    base,
    torch_dtype=torch.float16,
    device_map="auto",
)
model = PeftModel.from_pretrained(model, adapter)

# Inference
prompt = "Generate YAML code for..."
# ... your inference code

Training Results

- Training Loss: ~1.25-1.30
- Validation Loss: ~1.50-1.55
- Training Time: ~8-9 hours
- Expected Score: 0.805-0.810 (YAML/XML improvement from 0.80195 baseline)

Strategy: YAML/XML Clean Expansion (Revised)

Key Insights from Expert AI Analysis:

The main causes of YAML/XML failures are:
1. Code fences (yaml, xml) appearing in output
2. Leading phrases ("Here's the output:", "Sure!") before output
3. u-10bei datasets' "Approach: → Output:" structure leaking into outputs

Solutions Implemented:
┌─────────────────────────────────────┬──────────────────────────────┬───────────────────────────────────┐
│               Problem               │           Solution           │              Effect               │
├─────────────────────────────────────┼──────────────────────────────┼───────────────────────────────────┤
│ u-10bei "Output:" structure         │ Extract only after "Output:" │ Removes explanation tendency      │
├─────────────────────────────────────┼──────────────────────────────┼───────────────────────────────────┤
│ Insufficient "Return ONLY" examples │ Boost daichira datasets 2x   │ Strengthens direct output pattern │
├─────────────────────────────────────┼──────────────────────────────┼───────────────────────────────────┤
│ YAML/XML too weak                   │ Boost YAML/XML 2x            │ More training data                │
└─────────────────────────────────────┴──────────────────────────────┴───────────────────────────────────┘
Expected Improvements:
┌────────┬──────────┬────────┬──────────────────────────┐
│ Format │ Baseline │ Target │         Strategy         │
├────────┼──────────┼────────┼──────────────────────────┤
│ YAML   │ 94.3%    │ 96-97% │ Clean u-10bei + 2x boost │
├────────┼──────────┼────────┼──────────────────────────┤
│ XML    │ 85.0%    │ 88-90% │ Clean u-10bei + 2x boost │
├────────┼──────────┼────────┼──────────────────────────┤
│ TOML   │ 100.0%   │ 100%   │ Maintain                 │
├────────┼──────────┼────────┼──────────────────────────┤
│ JSON   │ 96.0%    │ 96%+   │ Maintain                 │
├────────┼──────────┼────────┼──────────────────────────┤
│ CSV    │ 100.0%   │ 100%   │ Maintain                 │
└────────┴──────────┴────────┴──────────────────────────┘
License

Apache 2.0

---
Trained on Cleaned StructEval dataset (YAML/XML Boosted - Revised)
Learning Rate: 6e-6 (conservative setting)
Strategy: u-10bei Output extraction + daichira 2x + YAML/XML 2x
Based on expert AI analysis