DFveloper/gemma-4-26B-A4B-it-qat-q4_0-Claude-Opus-Distill-v2-GGUF

🌟 Lumen 3.1 Pulsar QAT × Claude Opus 4.6 Distill

This model combines the Quantization-Aware Trained Lumen 3.1 Pulsar checkpoint with the fine-tuning delta extracted from:

`TeichAI/gemma-4-26B-A4B-it-Claude-Opus-Distill-v2`

Rather than directly fine-tuning or quantizing the TeichAI model, its learned weight changes were extracted relative to its original base model and approximated as a LoRA adapter.

That adapter was then merged into the Lumen 3.1 Pulsar QAT checkpoint.

In short:

Original Gemma 4 Base
        │
        │ Fine-tuning
        ▼
TeichAI Claude Opus Distill
        │
        │ Weight Diff
        ▼
Extracted LoRA
        │
        │ Merge
        ▼
Lumen 3.1 Pulsar QAT
        │
        ▼
Final Model

The resulting model therefore retains Lumen 3.1 Pulsar as its primary checkpoint while incorporating part of the learned behavior introduced by the Claude Opus reasoning distillation.

🧬 Model Construction

The model was produced using a delta-to-LoRA transfer pipeline.

Stage 1 — Obtain the Fine-Tuning Delta

Let:

W_base = weights of the original Gemma 4 base model
W_ft   = weights of the TeichAI Claude Opus distilled model

The fine-tuning delta is:

ΔW = W_ft - W_base

This isolates the changes introduced by the Claude Opus reasoning fine-tuning from the underlying Gemma 4 weights.

Stage 2 — Approximate the Delta with LoRA

Instead of directly merging the full dense weight difference, the delta matrices were decomposed into low-rank representations:

ΔW ≈ B · A

where A and B form a LoRA adapter.

Conceptually:

TeichAI Fine-Tuned Model
        -
Original Base Model
        │
        ▼
Dense Fine-Tuning Delta
        │
        ▼
Low-Rank Decomposition
        │
        ▼
LoRA Adapter

This allows the learned fine-tuning direction to be transferred independently of the original full checkpoint.

Stage 3 — Merge into Lumen 3.1 Pulsar QAT

The extracted LoRA was then applied to the Lumen 3.1 Pulsar QAT checkpoint:

W_final = W_lumen_qat + ΔW_lora

where:

W_lumen_qat = Lumen 3.1 Pulsar QAT weights
ΔW_lora     = reconstructed Claude Opus distillation delta

This is fundamentally different from:

TeichAI Model
    ↓
QAT
    ↓
Quantized TeichAI Model

No such pipeline was used.

Instead:

TeichAI Distill
      │
      ▼
Extract learned delta
      │
      ▼
Compress delta into LoRA
      │
      ▼
Merge into Lumen 3.1 Pulsar QAT

⚡ Why This Approach?

Directly replacing Lumen 3.1 Pulsar with the TeichAI checkpoint would discard the training and QAT characteristics already present in Lumen.

The delta-transfer approach attempts to preserve the existing Lumen checkpoint while selectively importing useful behavior learned during a separate fine-tuning run.

This has several advantages.

Preserve the Lumen base

The final checkpoint remains fundamentally based on:

Lumen 3.1 Pulsar QAT

rather than the TeichAI model.

This preserves the majority of Lumen's existing:

• —model behavior,
• —training history,
• —reasoning characteristics,
• —QAT adaptation,
• —quantization robustness.

Transfer fine-tuning behavior

The extracted LoRA represents only the difference introduced by fine-tuning.

As a result, it can transfer portions of the Claude Opus distillation without copying the entire source model.

Maintain QAT characteristics

Because the destination checkpoint is already Quantization-Aware Trained, the merge is performed directly on top of the QAT-trained Lumen weights.

The final model should therefore be considered:

Lumen 3.1 Pulsar QAT with an externally extracted reasoning LoRA merged into it

rather than:

a QAT version of the TeichAI Claude Opus model

🧠 Source of the Reasoning Delta

The transferred LoRA originates from:

TeichAI/gemma-4-26B-A4B-it-Claude-Opus-Distill-v2

That model was fine-tuned using high-effort Claude reasoning data, including:

These datasets were not necessarily used to directly train this checkpoint.

Instead, their influence is inherited indirectly through the extracted fine-tuning delta.

This distinction is important:

Datasets
   ↓
TeichAI Fine-Tune
   ↓
Weight Changes
   ↓
LoRA Extraction
   ↓
Lumen 3.1 Pulsar QAT

🔬 Technical Overview

The complete pipeline can be represented as:

             ┌────────────────────────────────┐
             │ Original Gemma 4 Checkpoint  │
             └──────────────┬─────────────────┘
                            │
                            │ SFT
                            ▼
             ┌────────────────────────────────┐
             │ TeichAI Claude Opus Distill  │
             └──────────────┬─────────────────┘
                            │
          Compute parameter │ differences
                            ▼
                  ΔW = W_ft - W_base
                            │
                            │ Low-rank approximation
                            ▼
                   ┌──────────────────┐
                   │ Extracted LoRA  │
                   └───────┬──────────┘
                           │
                           │ Merge
                           ▼
              ┌─────────────────────────┐
              │ Lumen 3.1 Pulsar QAT  │
              └────────────┬────────────┘
                          │
                          ▼
              ┌──────────────────────────┐
              │      Final Model       │
              └──────────────────────────┘

🔹 Delta-LoRA Extraction

For a matrix-shaped parameter:

D = W_ft - W_base

the dense delta can be approximated using a truncated low-rank decomposition such as:

D ≈ U_r Σ_r V_rᵀ

which can then be represented in LoRA form as:

B = U_r Σ_r
A = V_rᵀ

giving:

BA ≈ D

Only the portion of the fine-tuning delta captured by the selected rank is transferred.

Therefore, this method is intentionally an approximation of the source model's training delta, not an exact reconstruction of the entire fine-tuned model.

⚠️ Important Notes

This is not a conventional LoRA fine-tune

The LoRA adapter used here was not trained directly on Lumen 3.1 Pulsar.

It was reconstructed from the parameter difference between another model and its original base.

Therefore, the resulting behavior can differ from both:

• —the original Lumen checkpoint, and
• —the original TeichAI Claude Opus distilled model.

LoRA transfer is not lossless

Because the dense fine-tuning delta is projected into a limited rank:

ΔW → Low-Rank ΔW

some information is discarded.

The amount retained depends on factors such as:

• —LoRA rank,
• —singular-value distribution,
• —target modules,
• —architecture compatibility,
• —merge scaling,
• —differences between source and destination checkpoints.

Architecture compatibility matters

Delta transfer assumes that corresponding tensors between the source base model, source fine-tuned model, and destination model remain structurally compatible.

The technique is most meaningful when:

shape(source tensor)
=
shape(base tensor)
=
shape(destination tensor)

for the transferred modules.

QAT status

The QAT component comes from Lumen 3.1 Pulsar QAT itself.

The extracted Claude Opus LoRA was derived from a separate fine-tuned model and subsequently merged into the QAT checkpoint.

Therefore the correct lineage is:

Lumen 3.1 Pulsar
      │
      ▼
     QAT
      │
      ▼
Lumen 3.1 Pulsar QAT
      │
      │ + Extracted Claude Opus LoRA
      ▼
Final Model

🌟 Intended Capabilities

The final model aims to combine characteristics from both sources.

From Lumen 3.1 Pulsar QAT:

• —general reasoning,
• —instruction following,
• —coding,
• —efficient inference,
• —quantization robustness,
• —Lumen-specific model behavior.

From the extracted Claude Opus distillation delta:

• —structured reasoning,
• —complex task decomposition,
• —coding-oriented reasoning,
• —analytical responses,
• —high-effort problem-solving behavior.

The exact degree to which each capability transfers depends on the extracted LoRA rank and the interaction between the source delta and the Lumen checkpoint.

📊 Evaluation

Benchmarks are currently being evaluated.

Particularly useful comparisons include:

Lumen 3.1 Pulsar QAT
vs.
Lumen 3.1 Pulsar QAT + Claude Opus Delta-LoRA
vs.
Original TeichAI Claude Opus Distill

Recommended metrics include:

Because this is a cross-checkpoint delta transfer, direct evaluation against the unmodified Lumen checkpoint is particularly important.

🛠️ Usage

Use the same inference stack and chat template as the corresponding Lumen 3.1 Pulsar checkpoint.

Example with Transformers:

from transformers import AutoProcessor, AutoModelForCausalLM

MODEL_ID = "YOUR_MODEL_ID"

processor = AutoProcessor.from_pretrained(MODEL_ID)

model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    dtype="auto",
    device_map="auto",
)

messages = [
    {
        "role": "user",
        "content": "Explain why low-rank adaptation can approximate a fine-tuning delta."
    }
]

text = processor.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)

inputs = processor(
    text=text,
    return_tensors="pt",
).to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=1024,
)

response = processor.decode(
    outputs[0][inputs["input_ids"].shape[-1]:],
    skip_special_tokens=False,
)

print(response)

🙏 Acknowledgements

• —Google — for the Gemma architecture.
• —TeichAI — for releasing the Claude Opus reasoning-distilled Gemma checkpoint used as the source of the fine-tuning delta.
• —Crownelius — for the Opus reasoning dataset used in the source model.
• —Unsloth — for efficient Gemma fine-tuning tooling and ecosystem support.

📖 Model Lineage

                         SOURCE BRANCH

unsloth/gemma-4-26B-A4B-it
             │
             ▼
Claude Opus Reasoning Fine-Tuning
             │
             ▼
TeichAI/gemma-4-26B-A4B-it-Claude-Opus-Distill-v2
             │
             │ subtract original checkpoint
             ▼
      Fine-Tuning Delta
             │
             │ low-rank decomposition
             ▼
       Extracted LoRA
             │
             │
             └────────────────────┐
                                  │
                         DESTINATION BRANCH
                                  │
                       Lumen 3.1 Pulsar
                                  │
                                  ▼
                                 QAT
                                  │
                                  ▼
                       Lumen 3.1 Pulsar QAT
                                  │
                                  │ + LoRA Merge
                                  ▼
                              Final Model

Citation

If you use this model, please also credit the upstream checkpoints and datasets from which the transferred training delta was derived.