trd92/qwen25-moe-depth-parametric

MoE Depth + Parametric Activation Model Deployment

This directory contains the production-ready model with MoE Depth Schedule and Parametric Activation interventions applied to Qwen2.5-Coder-1.5B-Instruct.

Model Details

• —Base Model: Qwen/Qwen2.5-Coder-1.5B-Instruct
• —Interventions: MoE Depth Schedule + Parametric Activation
• —Parameters: 1543.75M total (0.04M additional from interventions)
• —Latency: 20.79s (vs 21.36s baseline)
• —Config: 60% acceptance rate for MoE routing

Directory Contents

├── config.json                    # Model configuration
├── model.safetensors              # Model weights (including trained intervention weights)
├── tokenizer.json                 # Tokenizer configuration
├── tokenizer_config.json          # Tokenizer settings
├── merges.txt                     # BPE merges
├── vocab.json                     # Vocabulary
├── special_tokens_map.json        # Special tokens
├── generation_config.json         # Generation settings
├── added_tokens.json              # Added tokens
├── chat_template.jinja            # Chat template
├── deploy_model.py                # Model loader for inference
├── test_model.py                  # Test script to verify deployment
└── README.md                      # This file

Quick Start

1. Test the Model

cd neuroanatomy/finetuned/qwen25_moe_depth_parametric_deploy
python test_model.py

2. Load for Inference

from deploy_model import load_intervened_model

model, tokenizer = load_intervened_model()

# Generate code
inputs = tokenizer("def fibonacci(n: int) -> int:", return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=100, do_sample=False)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

3. Custom Weights Path

from deploy_model import load_intervened_model

model, tokenizer = load_intervened_model(weights_path="path/to/model.safetensors")

Architecture

The model includes:

1. 1.MoE Depth Schedule: Routes tokens through different MLP experts based on layer depth
2. 2.Early layers (2, 5, 8): Handle low-level syntax
3. 3.Middle layers (11, 14, 17): Handle semantic structure
4. 4.Late layers (20, 23, 26): Handle high-level logic

1. 1.Parametric Activation: Learnable activation functions that adapt during training
2. 2.Applied to layers 2, 5, 8, 11, 14, 17, 20, 23, 26
3. 3.Initializes with GELU activation
4. 4.Per-channel activation mixing

Training Details

• —Training Steps: 20 steps (demo)
• —Freeze Base: True (only intervention weights are trained)
• —Learning Rate: 0.001
• —Batch Size: 1
• —Gradient Accumulation: 4

Performance

Requirements

• —Python 3.10+
• —PyTorch 2.0+
• —Transformers 4.30+
• —Safetensors
• —CUDA (recommended)

Notes

• —The model is saved with model.safetensors format
• —When loading, the base Qwen2 model is loaded first, then interventions are applied, and finally the trained weights are loaded
• —The strict=False parameter in load_state_dict() allows for missing keys (some base weights may not be in the trained state dict)

License

This model is based on Qwen2.5-Coder-1.5B-Instruct and follows the same license terms.