OthmaneBen/dfe-domain-expert-r96

license: other licensename: qwen basemodel: Qwen/Qwen3.5-9B library_name: peft tags: [dora, peft, adapter, dfe-stacking, qwen3.5] --------------------------------------------------

Domain Foundation Expert (DoRA r=96) | DFE-Stacking release

DoRA adapter from the paper "Fine-Tuning Shifts Form Before Competence". This is one of 12 released adapters covering all experimental conditions and seeds from a matched-arm ablation on Qwen 3.5 9B (Gated-DeltaNet/attention hybrid).

Usage

from transformers import AutoModelForCausalLM
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3.5-9B",
    torch_dtype="bfloat16"
)
model = PeftModel.from_pretrained(base, "OthmaneBen/dfe-domain-expert-r96")

# For stacked task experts, first load and merge the domain expert:
# base = PeftModel.from_pretrained(
#     base,
#     "OthmaneBen/dfe-domain-expert-r96"
# ).merge_and_unload()

Generation warning: the base checkpoint's eos_token_id does not include the chat end-of-turn token. Pass the full stop list or use the generation_config.json included in this repo. Merge before batch generation because unmerged DoRA is about 10x slower.

Findings supported by this adapter, as described in the paper: no seen-task benefit from stacking; a deterministic version-formatting coercion out of distribution (flat 72%, stacked 38 to 45%, joint 0%); and no genuine capability loss detected on any audited instrument.

═══════════════════════════════════════════════════════════════════