CoolFace
Modelpublic

yuk1chan/qwen3-4b-structeval-strategy7-baseline-yamlxml-lr6e-6

sourceHugging Faceapache-2.0updated 7mo agoView on Hugging Face
0likes12downloads
Model Card

qwen3-4b-structeval-strategy7-baseline-yamlxml-lr6e-6

This repository provides a LoRA adapter fine-tuned from Qwen/Qwen3-4B-Instruct-2507 using QLoRA (4-bit, Unsloth).

Training Objective

This adapter is trained to improve structured output accuracy (JSON / YAML / XML / TOML / CSV).

Strategy 7: Baseline Return + YAML/XML Pure Increase πŸ”₯

Training Configuration

  • β€”Base model: Qwen/Qwen3-4B-Instruct-2507
  • β€”Method: QLoRA (4-bit)
  • β€”Max sequence length: 1024
  • β€”Epochs: 1
  • β€”Learning rate: 6e-06
  • β€”LoRA: r=16, alpha=32

Dataset: Baseline Return + YAML/XML Boost

Strategy 7: Reverting to Baseline's TOML 100% Success

Baseline Success (0.80195):

  • β€”TOML: 100% βœ…
  • β€”YAML: 91.4%
  • β€”XML: 78.0%
  • β€”TOML ratio: 14%

Problem with Strategy 2 Revised (0.82286):

  • β€”TOML: 76.0% ❌ (down from 100%)
  • β€”YAML: 97.1% βœ…
  • β€”XML: 90.0% βœ…
  • β€”TOML ratio: 10%

Strategy 3 & 4 Failure:

  • β€”TOML ratio 14% did NOT reproduce Baseline's TOML 100%
  • β€”Something else was missing

Strategy 7 Solution:

1. Revert to Baseline's daichira level

  • β€”daichiraη³»: 1.0x (Baseline level, no boost)
  • β€”Avoid daichira reduction that harmed TOML

2. Pure YAML/XML increase

  • β€”u-10bei v2/v4/v5: 2.0x boost (YAML-rich)
  • β€”u-10bei base512/base: 1.3x boost (TOML-rich)
  • β€”u-10bei v2_short: 1.3x boost (TOML-rich)

3. Maintain TOML 14% ratio

  • β€”TOML absolute quantity: +30% (2,800 β†’ 3,640)
  • β€”TOML ratio: 14% (same as Baseline)
  • β€”Expected TOML recovery: 95-98%

Data Cleaning Pipeline:

  1. 1.CoT tags removal: <thinking>...</thinking> completely removed
  2. 2.Code fence removal: ```yaml, ``json, ``xml, ``toml, ```csv removed
  3. 3.Leading phrase removal: "Here's the output:", "Sure!" etc. removed
  4. 4.πŸ”₯ Output extraction: For u-10bei datasets, extract only content after "Output:" marker
  5. 5.Format validation: JSON/YAML/XML/TOML/CSV parsing validation
  6. 6.Deduplication: Exact duplicates removed

Format Distribution (Expected):

  • β€”YAML: ~13,000 (50%) πŸ”₯ (up from 35% in Baseline)
  • β€”XML: ~5,200 (20%) πŸ”₯ (up from 18% in Baseline)
  • β€”TOML: ~3,640 (14%) πŸ”₯ (same ratio as Baseline, +30% absolute)
  • β€”JSON: ~1,560 (6%)
  • β€”CSV: ~2,600 (10%)

Total: ~26,000 samples

Source Datasets with Boost Factors:

daichira系（1.0x - Baseline levelοΌ‰:

  • β€”daichira/structured-3k-mix-sft πŸ”₯ 1.0x (TOML 0%)
  • β€”daichira/structured-5k-mix-sft πŸ”₯ 1.0x (TOML 0%)
  • β€”daichira/structured-hard-sft-4k πŸ”₯ 1.0x (TOML 0%)

u-10bei系（YAML/XMLεΌ·εŒ–οΌ‰:

  • β€”u-10bei/structureddatawithcotdataset512v2 πŸ”₯ 2.0x (YAML-rich)
  • β€”u-10bei/structureddatawithcotdataset512v4 πŸ”₯ 2.0x (YAML-rich)
  • β€”u-10bei/structureddatawithcotdataset512v5 πŸ”₯ 2.0x (YAML-rich)
  • β€”u-10bei/structureddatawithcotdataset_512 πŸ”₯ 1.3x (TOML 20%)
  • β€”u-10bei/structureddatawithcotdataset πŸ”₯ 1.3x (TOML 17.6%)
  • β€”u-10bei/structureddatawithcotdataset_v2 πŸ”₯ 1.3x (TOML 15.9%)

Usage

python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base = "Qwen/Qwen3-4B-Instruct-2507"
adapter = "yuk1chan/qwen3-4b-structeval-strategy7-baseline-yamlxml-lr6e-6"

tokenizer = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(
    base,
    torch_dtype=torch.float16,
    device_map="auto",
)
model = PeftModel.from_pretrained(model, adapter)

# Inference
prompt = "Generate YAML code for..."
# ... your inference code

Training Results

  • β€”Training Loss: ~1.25-1.30
  • β€”Validation Loss: ~1.50-1.55
  • β€”Training Time: ~9-10 hours (T4 GPU)
  • β€”Expected Score: 0.840-0.860

Strategy: Baseline Return + YAML/XML Pure Increase

Key Insights:

Baseline's TOML 100% Success:

  • β€”TOML ratio: 14%
  • β€”TOML score: 100%
  • β€”This is the only time TOML achieved 100%

Why Strategy 2/3/4 Failed:

  • β€”Strategy 2: TOML ratio 10% β†’ TOML 76% (ratio too low)
  • β€”Strategy 3: TOML ratio 14-15% β†’ TOML ~70% (something else was wrong)
  • β€”Strategy 4: TOML ratio 11-12% β†’ TOML 0 samples (daichira reduction failed)

Strategy 7 Design:

AspectBaselineStrategy 2Strategy 7
daichira boost1.0x2.0x1.0x
TOML ratio14%10%14%
TOML absolute2,8002,6003,640
TOML score100%76%95-98%
YAML ratio35%45%50%
YAML score91.4%97.1%95-97%
Overall0.801950.822860.840-0.860

Expected Improvements:

  • β€”TOML recovery: 76% β†’ 95-98% (+19-22%)
  • β€”YAML maintained: 97.1% β†’ 95-97%
  • β€”XML: 90.0% β†’ 88-90%
  • β€”Overall: 0.82286 β†’ 0.840-0.860 (+1.7-3.7%)

Risk Analysis:

  • β€”Risk: Baseline's TOML 100% may not be reproducible
  • β€”Mitigation: Maintain TOML 14% ratio, daichira 1.0x
  • β€”Expected: High chance of TOML recovery

License

Apache 2.0

Trained on Baseline Return + YAML/XML Boost StructEval dataset Learning Rate: 6e-6 (proven setting from 0.82286) Strategy: daichira 1.0x (Baseline) + u-10bei 1.3-2.0x (YAML/XML boost) TOML ratio: 14% (same as Baseline)