haielab/Qwen3.6-27B-LoRA-fermipy-clean

Qwen3.6-27B-LoRA-fermipy (clean training, v2)

A LoRA adapter that fine-tunes Qwen/Qwen3.6-27B-FP8 to generate FermiPy YAML configurations and Python analysis scripts from natural-language descriptions of Fermi-LAT gamma-ray analyses.

v2 note. The earlier version of this adapter (`haielab/Qwen3.6-27B-LoRA-fermipy`) was trained on a pool that included examples mentioning the same three sources used as held-out evaluation targets in the parent paper (Mrk\,421, Vela, Crab). This v2 was trained with an explicit contamination filter that drops every such example (16 of 118 dropped, 14% of the pool). All metrics in this card are honest held-out numbers.

Quick start with vLLM

vllm serve Qwen/Qwen3.6-27B-FP8 \
    --enable-lora --max-loras 1 --max-lora-rank 32 \
    --lora-modules fermipy=ai4helab/Qwen3.6-27B-LoRA-fermipy-clean \
    --port 8000

from openai import OpenAI

SYSTEM_PROMPT = """You are a FermiPy expert. Given a natural-language description of a Fermi-LAT gamma-ray analysis, generate two artifacts:
1. A YAML configuration file for FermiPy's GTAnalysis.
2. A Python script that uses the FermiPy API to perform the analysis.

Return EXACTLY one fenced block of each, in this order, with no extra prose:

### YAML Configuration:

<yaml>

### Python Script:

<python>

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
resp = client.chat.completions.create(
    model="fermipy",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": "Generate FermiPy code for ..."},
    ],
    temperature=0.1, top_p=0.95, max_tokens=4096,
)
print(resp.choices[0].message.content)

The output always opens with ### YAML Configuration: followed by a fenced YAML block, then ### Python Script: with a fenced Python block. Downstream parsers should look for those fences.

Training details

• —Base model: Qwen/Qwen3.6-27B-FP8. The LoRA was trained on a one-off bf16 dequantization of this checkpoint; LoRA weights are dtype-independent and load on the FP8 base via vLLM.
• —LoRA: r=32, alpha=64, dropout=0.05, no bias adaptation.
• —Target modules (10 projections per layer × 64 layers, ~218 M trainable params, ≈0.8% of base):
• —16 standard self-attention layers: q_proj, k_proj, v_proj, o_proj
• —48 linear-attention (Mamba-style) layers: in_proj_qkv, in_proj_z, out_proj
• —64 MLP layers: gate_proj, up_proj, down_proj
• —Training data: 87 LLM-curated + 11 human-written examples after the contamination filter (98 total; 2 additional curated examples held out for in-loop validation).
• —Curriculum: Phase 1 — 6 epochs at lr=1e-4 on 87 curated; Phase 2 — 4 epochs at lr=2e-5 on 11 human. AdamW, cosine schedule, weight decay 0.01.
• —Sequence length: 4{,}096; effective batch size 8 (per-device 1 × grad-accum 8); gradient checkpointing on; bf16.
• —Hardware/wall-clock: 1× NVIDIA H100 80 GB; ~26 min phase 1, ~3 min phase 2.
• —Final losses: phase 1 = 0.4546, phase 2 = 0.2782.

The companion file training_metrics.json in this repo has the full hyperparameter snapshot.

Held-out evaluation

Evaluated through the four-level Validator Agent pipeline of the parent paper on Mrk\,421, Vela, and Crab.

• —Strict API recall uses the gta.<call>( regex used throughout the parent paper. The Mrk 421 generation used gt. as the variable name, deflating its strict score; the relaxed column matches gta?\. for that case.
• —L3 / L4 success is determined by actually running gta.setup() and gta.optimize() + gta.fit() against real Fermi-LAT data and checking 4FGL flux consistency.
• —Crab L3 failure was a persistent NDSKEYS IRF-mismatch error that the deterministic repair pipeline did not resolve in 5 iterations. This is the same failure mode that affects other models on Crab in the parent paper.

The companion file eval_results.json has the full per-target output (generated YAML, repaired YAML, repair logs, validation results, L4 fit results).

How this compares

In the data-scarce regime of this paper (≤120 curated examples), prompt-engineering the larger Qwen3.5-35B-A3B MoE (no fine-tune) is the better cost–quality tradeoff:

The LoRA matches the MoE's per-approach Level 4 rate but underperforms on parameter accuracy. Use this adapter when the deployment scenario already serves Qwen3.6-27B-FP8 and a task-specific adapter is operationally convenient.

License

Apache 2.0, inherited from the base model.